Note: This is test shard 1 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AdminCliTest
[ RUN ] AdminCliTest.RebalancerReportOnly
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 14:08:44.667608 23538 test_util.cc:268] Using random seed: 105267910
I20240501 14:08:44.674047 23538 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:08:44.674105 23538 ts_itest-base.cc:113] --------------
I20240501 14:08:44.674129 23538 ts_itest-base.cc:114] 5 tablet servers
I20240501 14:08:44.674149 23538 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:08:44.674163 23538 ts_itest-base.cc:116] --------------
2024-05-01T14:08:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:08:44Z Disabled control of system clock
I20240501 14:08:44.683106 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.252.190:41609
--webserver_interface=127.22.252.190
--webserver_port=0
--builtin_ntp_servers=127.22.252.148:46725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.252.190:41609 with env {}
W20240501 14:08:44.782696 23550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:44.782917 23550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:44.782990 23550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:44.785864 23550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:08:44.785917 23550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:44.785948 23550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:08:44.785990 23550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:08:44.789542 23550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:46725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.252.190:41609
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.252.190:41609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.252.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.23550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:44.790355 23550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:44.791461 23550 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:44.798717 23556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:44.799472 23550 server_base.cc:1034] running on GCE node
W20240501 14:08:44.799965 23555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:44.800789 23558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:44.801211 23550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:44.802004 23550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:44.803145 23550 hybrid_clock.cc:648] HybridClock initialized: now 1714572524803136 us; error 22 us; skew 500 ppm
I20240501 14:08:44.804855 23550 webserver.cc:460] Webserver started at http://127.22.252.190:44339/ using document root <none> and password file <none>
I20240501 14:08:44.805399 23550 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:44.805483 23550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:44.805698 23550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:44.807268 23550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d707ce03de124858bf3bb325aea42fea"
format_stamp: "Formatted at 2024-05-01 14:08:44 on dist-test-slave-bkzb"
I20240501 14:08:44.807713 23550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d707ce03de124858bf3bb325aea42fea"
format_stamp: "Formatted at 2024-05-01 14:08:44 on dist-test-slave-bkzb"
I20240501 14:08:44.810920 23550 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:08:44.812949 23564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:44.813889 23550 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:44.813993 23550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d707ce03de124858bf3bb325aea42fea"
format_stamp: "Formatted at 2024-05-01 14:08:44 on dist-test-slave-bkzb"
I20240501 14:08:44.814077 23550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:44.847127 23550 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:44.847823 23550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:44.847993 23550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:44.854346 23550 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.190:41609
I20240501 14:08:44.854357 23616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.190:41609 every 8 connection(s)
I20240501 14:08:44.855266 23550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:08:44.857782 23617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:44.858575 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 23550
I20240501 14:08:44.858685 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:08:44.863042 23617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea: Bootstrap starting.
I20240501 14:08:44.864907 23617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:44.865541 23617 log.cc:824] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:44.866969 23617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea: No bootstrap required, opened a new log
I20240501 14:08:44.869349 23617 raft_consensus.cc:348] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d707ce03de124858bf3bb325aea42fea" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 41609 } }
I20240501 14:08:44.869516 23617 raft_consensus.cc:374] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:44.869580 23617 raft_consensus.cc:729] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d707ce03de124858bf3bb325aea42fea, State: Initialized, Role: FOLLOWER
I20240501 14:08:44.869966 23617 consensus_queue.cc:260] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d707ce03de124858bf3bb325aea42fea" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 41609 } }
I20240501 14:08:44.870092 23617 raft_consensus.cc:388] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:44.870152 23617 raft_consensus.cc:482] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:44.870265 23617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:44.871179 23617 raft_consensus.cc:504] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d707ce03de124858bf3bb325aea42fea" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 41609 } }
I20240501 14:08:44.871501 23617 leader_election.cc:302] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d707ce03de124858bf3bb325aea42fea; no voters:
I20240501 14:08:44.871767 23617 leader_election.cc:288] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:44.871882 23622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:44.872110 23622 raft_consensus.cc:686] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea [term 1 LEADER]: Becoming Leader. State: Replica: d707ce03de124858bf3bb325aea42fea, State: Running, Role: LEADER
I20240501 14:08:44.872442 23622 consensus_queue.cc:237] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d707ce03de124858bf3bb325aea42fea" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 41609 } }
I20240501 14:08:44.872895 23617 sys_catalog.cc:564] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:08:44.874008 23623 sys_catalog.cc:455] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d707ce03de124858bf3bb325aea42fea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d707ce03de124858bf3bb325aea42fea" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 41609 } } }
I20240501 14:08:44.874138 23623 sys_catalog.cc:458] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:44.874485 23631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:08:44.874452 23624 sys_catalog.cc:455] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea [sys.catalog]: SysCatalogTable state changed. Reason: New leader d707ce03de124858bf3bb325aea42fea. Latest consensus state: current_term: 1 leader_uuid: "d707ce03de124858bf3bb325aea42fea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d707ce03de124858bf3bb325aea42fea" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 41609 } } }
I20240501 14:08:44.874660 23624 sys_catalog.cc:458] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:44.876838 23631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:08:44.881702 23631 catalog_manager.cc:1347] Generated new cluster ID: fdcca4233ab5448280287f415cf02ab5
I20240501 14:08:44.881767 23631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:08:44.915033 23631 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:08:44.915974 23631 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:08:44.922935 23631 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d707ce03de124858bf3bb325aea42fea: Generated new TSK 0
I20240501 14:08:44.923593 23631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:08:44.928177 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.129:0
--local_ip_for_outbound_sockets=127.22.252.129
--webserver_interface=127.22.252.129
--webserver_port=0
--tserver_master_addrs=127.22.252.190:41609
--builtin_ntp_servers=127.22.252.148:46725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240501 14:08:45.033453 23641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:45.033663 23641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:45.033703 23641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:45.036684 23641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:45.036779 23641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.129
I20240501 14:08:45.040295 23641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:46725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.252.129
--webserver_port=0
--tserver_master_addrs=127.22.252.190:41609
--never_fsync=true
--heap_profile_path=/tmp/kudu.23641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.129
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:45.041097 23641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:45.042217 23641 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:45.050587 23649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:45.051079 23647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:45.051260 23646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:45.052142 23641 server_base.cc:1034] running on GCE node
I20240501 14:08:45.052500 23641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:45.052965 23641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:45.054119 23641 hybrid_clock.cc:648] HybridClock initialized: now 1714572525054098 us; error 34 us; skew 500 ppm
I20240501 14:08:45.055678 23641 webserver.cc:460] Webserver started at http://127.22.252.129:40765/ using document root <none> and password file <none>
I20240501 14:08:45.056195 23641 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:45.056280 23641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:45.056470 23641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:45.058033 23641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "98be687a46fa422c82f442abc1599f14"
format_stamp: "Formatted at 2024-05-01 14:08:45 on dist-test-slave-bkzb"
I20240501 14:08:45.058495 23641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "98be687a46fa422c82f442abc1599f14"
format_stamp: "Formatted at 2024-05-01 14:08:45 on dist-test-slave-bkzb"
I20240501 14:08:45.061798 23641 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240501 14:08:45.063774 23655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:45.064723 23641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 14:08:45.064827 23641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "98be687a46fa422c82f442abc1599f14"
format_stamp: "Formatted at 2024-05-01 14:08:45 on dist-test-slave-bkzb"
I20240501 14:08:45.064915 23641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:45.101789 23641 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:45.102468 23641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:45.102635 23641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:45.103168 23641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:45.104060 23641 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:45.104111 23641 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:45.104171 23641 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:45.104202 23641 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:45.112485 23641 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.129:38907
I20240501 14:08:45.112516 23768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.129:38907 every 8 connection(s)
I20240501 14:08:45.113413 23641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:08:45.113834 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 23641
I20240501 14:08:45.113938 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:08:45.116178 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.130:0
--local_ip_for_outbound_sockets=127.22.252.130
--webserver_interface=127.22.252.130
--webserver_port=0
--tserver_master_addrs=127.22.252.190:41609
--builtin_ntp_servers=127.22.252.148:46725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 14:08:45.122538 23769 heartbeater.cc:347] Connected to a master server at 127.22.252.190:41609
I20240501 14:08:45.122788 23769 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:45.123487 23769 heartbeater.cc:511] Master 127.22.252.190:41609 requested a full tablet report, sending...
I20240501 14:08:45.124931 23579 ts_manager.cc:194] Registered new tserver with Master: 98be687a46fa422c82f442abc1599f14 (127.22.252.129:38907)
I20240501 14:08:45.126081 23579 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.129:33547
W20240501 14:08:45.220760 23772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:45.220942 23772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:45.220981 23772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:45.223803 23772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:45.223891 23772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.130
I20240501 14:08:45.227377 23772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:46725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.252.130
--webserver_port=0
--tserver_master_addrs=127.22.252.190:41609
--never_fsync=true
--heap_profile_path=/tmp/kudu.23772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.130
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:45.228191 23772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:45.229276 23772 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:45.237426 23780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:45.238123 23778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:45.238921 23777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:45.239708 23772 server_base.cc:1034] running on GCE node
I20240501 14:08:45.240051 23772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:45.240604 23772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:45.241747 23772 hybrid_clock.cc:648] HybridClock initialized: now 1714572525241718 us; error 41 us; skew 500 ppm
I20240501 14:08:45.243438 23772 webserver.cc:460] Webserver started at http://127.22.252.130:35667/ using document root <none> and password file <none>
I20240501 14:08:45.243956 23772 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:45.244020 23772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:45.244215 23772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:45.245868 23772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1b8df2e61a254270bc0f8aafa0a6ab4b"
format_stamp: "Formatted at 2024-05-01 14:08:45 on dist-test-slave-bkzb"
I20240501 14:08:45.246317 23772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1b8df2e61a254270bc0f8aafa0a6ab4b"
format_stamp: "Formatted at 2024-05-01 14:08:45 on dist-test-slave-bkzb"
I20240501 14:08:45.249511 23772 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:08:45.251587 23786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:45.252609 23772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:08:45.252700 23772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1b8df2e61a254270bc0f8aafa0a6ab4b"
format_stamp: "Formatted at 2024-05-01 14:08:45 on dist-test-slave-bkzb"
I20240501 14:08:45.252792 23772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:45.282133 23772 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:45.282763 23772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:45.282920 23772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:45.283460 23772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:45.284355 23772 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:45.284405 23772 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:45.284466 23772 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:45.284502 23772 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:45.293084 23772 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.130:41485
I20240501 14:08:45.293121 23899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.130:41485 every 8 connection(s)
I20240501 14:08:45.293963 23772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:08:45.301831 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 23772
I20240501 14:08:45.301931 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:08:45.302492 23900 heartbeater.cc:347] Connected to a master server at 127.22.252.190:41609
I20240501 14:08:45.302727 23900 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:45.303241 23900 heartbeater.cc:511] Master 127.22.252.190:41609 requested a full tablet report, sending...
I20240501 14:08:45.304004 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.131:0
--local_ip_for_outbound_sockets=127.22.252.131
--webserver_interface=127.22.252.131
--webserver_port=0
--tserver_master_addrs=127.22.252.190:41609
--builtin_ntp_servers=127.22.252.148:46725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 14:08:45.304291 23579 ts_manager.cc:194] Registered new tserver with Master: 1b8df2e61a254270bc0f8aafa0a6ab4b (127.22.252.130:41485)
I20240501 14:08:45.305022 23579 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.130:32925
W20240501 14:08:45.407724 23903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:45.407910 23903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:45.407956 23903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:45.410872 23903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:45.410967 23903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.131
I20240501 14:08:45.414357 23903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:46725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.252.131
--webserver_port=0
--tserver_master_addrs=127.22.252.190:41609
--never_fsync=true
--heap_profile_path=/tmp/kudu.23903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.131
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:45.415241 23903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:45.416335 23903 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:45.424661 23911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:45.424741 23909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:45.425354 23908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:45.426208 23903 server_base.cc:1034] running on GCE node
I20240501 14:08:45.426506 23903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:45.426961 23903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:45.428112 23903 hybrid_clock.cc:648] HybridClock initialized: now 1714572525428093 us; error 32 us; skew 500 ppm
I20240501 14:08:45.429811 23903 webserver.cc:460] Webserver started at http://127.22.252.131:38857/ using document root <none> and password file <none>
I20240501 14:08:45.430325 23903 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:45.430380 23903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:45.430554 23903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:45.432152 23903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "208ec2ad9e154cc8b32898026d089451"
format_stamp: "Formatted at 2024-05-01 14:08:45 on dist-test-slave-bkzb"
I20240501 14:08:45.432636 23903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "208ec2ad9e154cc8b32898026d089451"
format_stamp: "Formatted at 2024-05-01 14:08:45 on dist-test-slave-bkzb"
I20240501 14:08:45.435600 23903 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:08:45.437633 23917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:45.438544 23903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:45.438644 23903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "208ec2ad9e154cc8b32898026d089451"
format_stamp: "Formatted at 2024-05-01 14:08:45 on dist-test-slave-bkzb"
I20240501 14:08:45.438722 23903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:45.452539 23903 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:45.453053 23903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:45.453187 23903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:45.453647 23903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:45.454430 23903 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:45.454478 23903 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:45.454535 23903 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:45.454562 23903 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:45.462846 23903 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.131:38481
I20240501 14:08:45.462875 24030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.131:38481 every 8 connection(s)
I20240501 14:08:45.463621 23903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:08:45.469781 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 23903
I20240501 14:08:45.469870 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:08:45.471776 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.132:0
--local_ip_for_outbound_sockets=127.22.252.132
--webserver_interface=127.22.252.132
--webserver_port=0
--tserver_master_addrs=127.22.252.190:41609
--builtin_ntp_servers=127.22.252.148:46725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 14:08:45.472573 24031 heartbeater.cc:347] Connected to a master server at 127.22.252.190:41609
I20240501 14:08:45.472769 24031 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:45.473243 24031 heartbeater.cc:511] Master 127.22.252.190:41609 requested a full tablet report, sending...
I20240501 14:08:45.474206 23579 ts_manager.cc:194] Registered new tserver with Master: 208ec2ad9e154cc8b32898026d089451 (127.22.252.131:38481)
I20240501 14:08:45.474696 23579 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.131:57413
W20240501 14:08:45.574518 24034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:45.574739 24034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:45.574805 24034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:45.577838 24034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:45.577967 24034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.132
I20240501 14:08:45.581360 24034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:46725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.252.132
--webserver_port=0
--tserver_master_addrs=127.22.252.190:41609
--never_fsync=true
--heap_profile_path=/tmp/kudu.24034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.132
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:45.582113 24034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:45.583114 24034 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:45.591221 24042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:45.591292 24039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:45.592101 24040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:45.592819 24034 server_base.cc:1034] running on GCE node
I20240501 14:08:45.593256 24034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:45.593866 24034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:45.595018 24034 hybrid_clock.cc:648] HybridClock initialized: now 1714572525594989 us; error 44 us; skew 500 ppm
I20240501 14:08:45.597254 24034 webserver.cc:460] Webserver started at http://127.22.252.132:35565/ using document root <none> and password file <none>
I20240501 14:08:45.597934 24034 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:45.598011 24034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:45.598212 24034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:45.599926 24034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "f71978358716496d9fc9e2cc4de51e6d"
format_stamp: "Formatted at 2024-05-01 14:08:45 on dist-test-slave-bkzb"
I20240501 14:08:45.600412 24034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "f71978358716496d9fc9e2cc4de51e6d"
format_stamp: "Formatted at 2024-05-01 14:08:45 on dist-test-slave-bkzb"
I20240501 14:08:45.603504 24034 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240501 14:08:45.605772 24048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:45.606704 24034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:45.606797 24034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f71978358716496d9fc9e2cc4de51e6d"
format_stamp: "Formatted at 2024-05-01 14:08:45 on dist-test-slave-bkzb"
I20240501 14:08:45.606885 24034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:45.620849 24034 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:45.621480 24034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:45.621629 24034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:45.622174 24034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:45.622995 24034 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:45.623040 24034 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:45.623077 24034 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:45.623100 24034 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:45.631292 24034 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.132:39359
I20240501 14:08:45.631335 24161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.132:39359 every 8 connection(s)
I20240501 14:08:45.632206 24034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:08:45.637779 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 24034
I20240501 14:08:45.637888 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:08:45.640167 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.133:0
--local_ip_for_outbound_sockets=127.22.252.133
--webserver_interface=127.22.252.133
--webserver_port=0
--tserver_master_addrs=127.22.252.190:41609
--builtin_ntp_servers=127.22.252.148:46725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 14:08:45.640283 24162 heartbeater.cc:347] Connected to a master server at 127.22.252.190:41609
I20240501 14:08:45.640486 24162 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:45.641072 24162 heartbeater.cc:511] Master 127.22.252.190:41609 requested a full tablet report, sending...
I20240501 14:08:45.642141 23579 ts_manager.cc:194] Registered new tserver with Master: f71978358716496d9fc9e2cc4de51e6d (127.22.252.132:39359)
I20240501 14:08:45.642781 23579 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.132:46575
W20240501 14:08:45.742707 24165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:45.742913 24165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:45.742977 24165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:45.745929 24165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:45.746018 24165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.133
I20240501 14:08:45.749504 24165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:46725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.252.133
--webserver_port=0
--tserver_master_addrs=127.22.252.190:41609
--never_fsync=true
--heap_profile_path=/tmp/kudu.24165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.133
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:45.750324 24165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:45.751449 24165 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:45.759120 24171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:45.759845 24170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:45.760423 24173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:45.761308 24165 server_base.cc:1034] running on GCE node
I20240501 14:08:45.761632 24165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:45.762113 24165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:45.763270 24165 hybrid_clock.cc:648] HybridClock initialized: now 1714572525763263 us; error 32 us; skew 500 ppm
I20240501 14:08:45.764946 24165 webserver.cc:460] Webserver started at http://127.22.252.133:45793/ using document root <none> and password file <none>
I20240501 14:08:45.765487 24165 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:45.765547 24165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:45.765727 24165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:45.767315 24165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "7ca753dec4994167844449d2ac755d5d"
format_stamp: "Formatted at 2024-05-01 14:08:45 on dist-test-slave-bkzb"
I20240501 14:08:45.767760 24165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "7ca753dec4994167844449d2ac755d5d"
format_stamp: "Formatted at 2024-05-01 14:08:45 on dist-test-slave-bkzb"
I20240501 14:08:45.770970 24165 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:08:45.773231 24179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:45.774206 24165 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:08:45.774308 24165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7ca753dec4994167844449d2ac755d5d"
format_stamp: "Formatted at 2024-05-01 14:08:45 on dist-test-slave-bkzb"
I20240501 14:08:45.774384 24165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:45.796990 24165 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:45.797573 24165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:45.797717 24165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:45.798216 24165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:45.799088 24165 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:45.799134 24165 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:45.799193 24165 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:45.799219 24165 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:45.807626 24165 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.133:46467
I20240501 14:08:45.807693 24292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.133:46467 every 8 connection(s)
I20240501 14:08:45.808565 24165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:08:45.816210 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 24165
I20240501 14:08:45.816355 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:08:45.817257 24293 heartbeater.cc:347] Connected to a master server at 127.22.252.190:41609
I20240501 14:08:45.817463 24293 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:45.818027 24293 heartbeater.cc:511] Master 127.22.252.190:41609 requested a full tablet report, sending...
I20240501 14:08:45.818964 23579 ts_manager.cc:194] Registered new tserver with Master: 7ca753dec4994167844449d2ac755d5d (127.22.252.133:46467)
I20240501 14:08:45.819509 23579 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.133:53219
I20240501 14:08:45.830531 23538 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 14:08:45.841382 23579 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48754:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 14:08:45.859625 23703 tablet_service.cc:1467] Processing CreateTablet for tablet 2e022a0a35dd4e1d8e88bf9105c240cb (DEFAULT_TABLE table=TestTable [id=8d5486289d37491b964832a00d57ce76]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:45.859623 24227 tablet_service.cc:1467] Processing CreateTablet for tablet 2e022a0a35dd4e1d8e88bf9105c240cb (DEFAULT_TABLE table=TestTable [id=8d5486289d37491b964832a00d57ce76]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:45.860653 23703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e022a0a35dd4e1d8e88bf9105c240cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:45.860653 24227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e022a0a35dd4e1d8e88bf9105c240cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:45.864621 23834 tablet_service.cc:1467] Processing CreateTablet for tablet 2e022a0a35dd4e1d8e88bf9105c240cb (DEFAULT_TABLE table=TestTable [id=8d5486289d37491b964832a00d57ce76]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:45.865502 23834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e022a0a35dd4e1d8e88bf9105c240cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:45.867264 24310 tablet_bootstrap.cc:492] T 2e022a0a35dd4e1d8e88bf9105c240cb P 98be687a46fa422c82f442abc1599f14: Bootstrap starting.
I20240501 14:08:45.868039 24311 tablet_bootstrap.cc:492] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d: Bootstrap starting.
I20240501 14:08:45.869395 24310 tablet_bootstrap.cc:654] T 2e022a0a35dd4e1d8e88bf9105c240cb P 98be687a46fa422c82f442abc1599f14: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:45.869786 24311 tablet_bootstrap.cc:654] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:45.870178 24310 log.cc:824] T 2e022a0a35dd4e1d8e88bf9105c240cb P 98be687a46fa422c82f442abc1599f14: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:45.870379 24311 log.cc:824] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:45.871717 24310 tablet_bootstrap.cc:492] T 2e022a0a35dd4e1d8e88bf9105c240cb P 98be687a46fa422c82f442abc1599f14: No bootstrap required, opened a new log
I20240501 14:08:45.871775 24311 tablet_bootstrap.cc:492] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d: No bootstrap required, opened a new log
I20240501 14:08:45.871860 24310 ts_tablet_manager.cc:1367] T 2e022a0a35dd4e1d8e88bf9105c240cb P 98be687a46fa422c82f442abc1599f14: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240501 14:08:45.871927 24311 ts_tablet_manager.cc:1367] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:08:45.872013 24313 tablet_bootstrap.cc:492] T 2e022a0a35dd4e1d8e88bf9105c240cb P 1b8df2e61a254270bc0f8aafa0a6ab4b: Bootstrap starting.
I20240501 14:08:45.873965 24313 tablet_bootstrap.cc:654] T 2e022a0a35dd4e1d8e88bf9105c240cb P 1b8df2e61a254270bc0f8aafa0a6ab4b: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:45.874401 24310 raft_consensus.cc:348] T 2e022a0a35dd4e1d8e88bf9105c240cb P 98be687a46fa422c82f442abc1599f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b8df2e61a254270bc0f8aafa0a6ab4b" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 41485 } } peers { permanent_uuid: "98be687a46fa422c82f442abc1599f14" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 38907 } } peers { permanent_uuid: "7ca753dec4994167844449d2ac755d5d" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46467 } }
I20240501 14:08:45.874553 24310 raft_consensus.cc:374] T 2e022a0a35dd4e1d8e88bf9105c240cb P 98be687a46fa422c82f442abc1599f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:45.874609 24310 raft_consensus.cc:729] T 2e022a0a35dd4e1d8e88bf9105c240cb P 98be687a46fa422c82f442abc1599f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98be687a46fa422c82f442abc1599f14, State: Initialized, Role: FOLLOWER
I20240501 14:08:45.874634 24313 log.cc:824] T 2e022a0a35dd4e1d8e88bf9105c240cb P 1b8df2e61a254270bc0f8aafa0a6ab4b: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:45.874969 24310 consensus_queue.cc:260] T 2e022a0a35dd4e1d8e88bf9105c240cb P 98be687a46fa422c82f442abc1599f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b8df2e61a254270bc0f8aafa0a6ab4b" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 41485 } } peers { permanent_uuid: "98be687a46fa422c82f442abc1599f14" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 38907 } } peers { permanent_uuid: "7ca753dec4994167844449d2ac755d5d" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46467 } }
I20240501 14:08:45.875759 24310 ts_tablet_manager.cc:1398] T 2e022a0a35dd4e1d8e88bf9105c240cb P 98be687a46fa422c82f442abc1599f14: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:08:45.876057 23769 heartbeater.cc:503] Master 127.22.252.190:41609 was elected leader, sending a full tablet report...
I20240501 14:08:45.876142 24313 tablet_bootstrap.cc:492] T 2e022a0a35dd4e1d8e88bf9105c240cb P 1b8df2e61a254270bc0f8aafa0a6ab4b: No bootstrap required, opened a new log
I20240501 14:08:45.876314 24313 ts_tablet_manager.cc:1367] T 2e022a0a35dd4e1d8e88bf9105c240cb P 1b8df2e61a254270bc0f8aafa0a6ab4b: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:08:45.876600 24311 raft_consensus.cc:348] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b8df2e61a254270bc0f8aafa0a6ab4b" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 41485 } } peers { permanent_uuid: "98be687a46fa422c82f442abc1599f14" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 38907 } } peers { permanent_uuid: "7ca753dec4994167844449d2ac755d5d" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46467 } }
I20240501 14:08:45.876777 24311 raft_consensus.cc:374] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:45.876866 24311 raft_consensus.cc:729] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ca753dec4994167844449d2ac755d5d, State: Initialized, Role: FOLLOWER
I20240501 14:08:45.877321 24311 consensus_queue.cc:260] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b8df2e61a254270bc0f8aafa0a6ab4b" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 41485 } } peers { permanent_uuid: "98be687a46fa422c82f442abc1599f14" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 38907 } } peers { permanent_uuid: "7ca753dec4994167844449d2ac755d5d" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46467 } }
I20240501 14:08:45.877935 24293 heartbeater.cc:503] Master 127.22.252.190:41609 was elected leader, sending a full tablet report...
I20240501 14:08:45.878123 24311 ts_tablet_manager.cc:1398] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 14:08:45.879019 24313 raft_consensus.cc:348] T 2e022a0a35dd4e1d8e88bf9105c240cb P 1b8df2e61a254270bc0f8aafa0a6ab4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b8df2e61a254270bc0f8aafa0a6ab4b" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 41485 } } peers { permanent_uuid: "98be687a46fa422c82f442abc1599f14" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 38907 } } peers { permanent_uuid: "7ca753dec4994167844449d2ac755d5d" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46467 } }
I20240501 14:08:45.879253 24313 raft_consensus.cc:374] T 2e022a0a35dd4e1d8e88bf9105c240cb P 1b8df2e61a254270bc0f8aafa0a6ab4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:45.879333 24313 raft_consensus.cc:729] T 2e022a0a35dd4e1d8e88bf9105c240cb P 1b8df2e61a254270bc0f8aafa0a6ab4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b8df2e61a254270bc0f8aafa0a6ab4b, State: Initialized, Role: FOLLOWER
I20240501 14:08:45.879722 24313 consensus_queue.cc:260] T 2e022a0a35dd4e1d8e88bf9105c240cb P 1b8df2e61a254270bc0f8aafa0a6ab4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b8df2e61a254270bc0f8aafa0a6ab4b" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 41485 } } peers { permanent_uuid: "98be687a46fa422c82f442abc1599f14" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 38907 } } peers { permanent_uuid: "7ca753dec4994167844449d2ac755d5d" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46467 } }
I20240501 14:08:45.880491 24313 ts_tablet_manager.cc:1398] T 2e022a0a35dd4e1d8e88bf9105c240cb P 1b8df2e61a254270bc0f8aafa0a6ab4b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:08:45.880704 23900 heartbeater.cc:503] Master 127.22.252.190:41609 was elected leader, sending a full tablet report...
W20240501 14:08:46.045279 23901 tablet.cc:2369] T 2e022a0a35dd4e1d8e88bf9105c240cb P 1b8df2e61a254270bc0f8aafa0a6ab4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:08:46.048403 24317 raft_consensus.cc:482] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:46.048599 24317 raft_consensus.cc:504] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b8df2e61a254270bc0f8aafa0a6ab4b" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 41485 } } peers { permanent_uuid: "98be687a46fa422c82f442abc1599f14" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 38907 } } peers { permanent_uuid: "7ca753dec4994167844449d2ac755d5d" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46467 } }
I20240501 14:08:46.049621 24317 leader_election.cc:288] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b8df2e61a254270bc0f8aafa0a6ab4b (127.22.252.130:41485), 98be687a46fa422c82f442abc1599f14 (127.22.252.129:38907)
I20240501 14:08:46.053705 23854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e022a0a35dd4e1d8e88bf9105c240cb" candidate_uuid: "7ca753dec4994167844449d2ac755d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b8df2e61a254270bc0f8aafa0a6ab4b" is_pre_election: true
I20240501 14:08:46.053743 23723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e022a0a35dd4e1d8e88bf9105c240cb" candidate_uuid: "7ca753dec4994167844449d2ac755d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98be687a46fa422c82f442abc1599f14" is_pre_election: true
I20240501 14:08:46.053946 23854 raft_consensus.cc:2454] T 2e022a0a35dd4e1d8e88bf9105c240cb P 1b8df2e61a254270bc0f8aafa0a6ab4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ca753dec4994167844449d2ac755d5d in term 0.
I20240501 14:08:46.053970 23723 raft_consensus.cc:2454] T 2e022a0a35dd4e1d8e88bf9105c240cb P 98be687a46fa422c82f442abc1599f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ca753dec4994167844449d2ac755d5d in term 0.
I20240501 14:08:46.054354 24180 leader_election.cc:302] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ca753dec4994167844449d2ac755d5d, 98be687a46fa422c82f442abc1599f14; no voters:
I20240501 14:08:46.054563 24317 raft_consensus.cc:2781] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:46.054631 24317 raft_consensus.cc:482] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:46.054670 24317 raft_consensus.cc:3037] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:46.055532 24317 raft_consensus.cc:504] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b8df2e61a254270bc0f8aafa0a6ab4b" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 41485 } } peers { permanent_uuid: "98be687a46fa422c82f442abc1599f14" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 38907 } } peers { permanent_uuid: "7ca753dec4994167844449d2ac755d5d" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46467 } }
I20240501 14:08:46.055815 24317 leader_election.cc:288] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d [CANDIDATE]: Term 1 election: Requested vote from peers 1b8df2e61a254270bc0f8aafa0a6ab4b (127.22.252.130:41485), 98be687a46fa422c82f442abc1599f14 (127.22.252.129:38907)
I20240501 14:08:46.056104 23854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e022a0a35dd4e1d8e88bf9105c240cb" candidate_uuid: "7ca753dec4994167844449d2ac755d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b8df2e61a254270bc0f8aafa0a6ab4b"
I20240501 14:08:46.056188 23723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e022a0a35dd4e1d8e88bf9105c240cb" candidate_uuid: "7ca753dec4994167844449d2ac755d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98be687a46fa422c82f442abc1599f14"
I20240501 14:08:46.056250 23854 raft_consensus.cc:3037] T 2e022a0a35dd4e1d8e88bf9105c240cb P 1b8df2e61a254270bc0f8aafa0a6ab4b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:46.056322 23723 raft_consensus.cc:3037] T 2e022a0a35dd4e1d8e88bf9105c240cb P 98be687a46fa422c82f442abc1599f14 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:46.057385 23854 raft_consensus.cc:2454] T 2e022a0a35dd4e1d8e88bf9105c240cb P 1b8df2e61a254270bc0f8aafa0a6ab4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ca753dec4994167844449d2ac755d5d in term 1.
I20240501 14:08:46.057391 23723 raft_consensus.cc:2454] T 2e022a0a35dd4e1d8e88bf9105c240cb P 98be687a46fa422c82f442abc1599f14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ca753dec4994167844449d2ac755d5d in term 1.
I20240501 14:08:46.057672 24180 leader_election.cc:302] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ca753dec4994167844449d2ac755d5d, 98be687a46fa422c82f442abc1599f14; no voters:
I20240501 14:08:46.057837 24317 raft_consensus.cc:2781] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:46.058043 24317 raft_consensus.cc:686] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d [term 1 LEADER]: Becoming Leader. State: Replica: 7ca753dec4994167844449d2ac755d5d, State: Running, Role: LEADER
I20240501 14:08:46.058334 24317 consensus_queue.cc:237] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b8df2e61a254270bc0f8aafa0a6ab4b" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 41485 } } peers { permanent_uuid: "98be687a46fa422c82f442abc1599f14" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 38907 } } peers { permanent_uuid: "7ca753dec4994167844449d2ac755d5d" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46467 } }
W20240501 14:08:46.059340 24294 tablet.cc:2369] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:08:46.061175 23579 catalog_manager.cc:5484] T 2e022a0a35dd4e1d8e88bf9105c240cb P 7ca753dec4994167844449d2ac755d5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ca753dec4994167844449d2ac755d5d (127.22.252.133). New cstate: current_term: 1 leader_uuid: "7ca753dec4994167844449d2ac755d5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b8df2e61a254270bc0f8aafa0a6ab4b" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 41485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98be687a46fa422c82f442abc1599f14" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 38907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ca753dec4994167844449d2ac755d5d" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46467 } health_report { overall_health: HEALTHY } } }
I20240501 14:08:46.085467 23538 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 14:08:46.086916 23538 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 98be687a46fa422c82f442abc1599f14 to finish bootstrapping
I20240501 14:08:46.091814 23538 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1b8df2e61a254270bc0f8aafa0a6ab4b to finish bootstrapping
I20240501 14:08:46.095778 23538 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 208ec2ad9e154cc8b32898026d089451 to finish bootstrapping
I20240501 14:08:46.100914 23538 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f71978358716496d9fc9e2cc4de51e6d to finish bootstrapping
I20240501 14:08:46.105679 23538 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7ca753dec4994167844449d2ac755d5d to finish bootstrapping
W20240501 14:08:46.114943 23770 tablet.cc:2369] T 2e022a0a35dd4e1d8e88bf9105c240cb P 98be687a46fa422c82f442abc1599f14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:08:46.284950 23834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:46.286487 23703 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:46.286742 23965 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:46.288429 24227 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:46.297405 24096 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:46.376602 23834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:46.377272 23965 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:46.379004 24227 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:46.379287 24096 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:46.399562 23703 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:46.423260 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 23641
I20240501 14:08:46.431002 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 23772
I20240501 14:08:46.436697 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 23903
I20240501 14:08:46.442119 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 24034
I20240501 14:08:46.447016 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 24165
I20240501 14:08:46.452142 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 23550
2024-05-01T14:08:46Z chronyd exiting
[ OK ] AdminCliTest.RebalancerReportOnly (1807 ms)
[----------] 1 test from AdminCliTest (1807 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/0
I20240501 14:08:46.474014 23538 test_util.cc:268] Using random seed: 107074330
I20240501 14:08:46.474727 23538 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:08:46.474772 23538 ts_itest-base.cc:113] --------------
I20240501 14:08:46.474783 23538 ts_itest-base.cc:114] 5 tablet servers
I20240501 14:08:46.474793 23538 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:08:46.474804 23538 ts_itest-base.cc:116] --------------
2024-05-01T14:08:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:08:46Z Disabled control of system clock
I20240501 14:08:46.482398 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.252.190:46861
--webserver_interface=127.22.252.190
--webserver_port=0
--builtin_ntp_servers=127.22.252.148:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.252.190:46861
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 14:08:46.581881 24398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:46.582063 24398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:46.582098 24398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:46.584822 24398 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:08:46.584874 24398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:08:46.584893 24398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:46.584910 24398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:08:46.584926 24398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:08:46.588382 24398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.252.190:46861
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.252.190:46861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.252.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.24398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:46.589205 24398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:46.590220 24398 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:46.597456 24406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:46.597699 24403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:46.598330 24404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:46.598968 24398 server_base.cc:1034] running on GCE node
I20240501 14:08:46.599431 24398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:46.600198 24398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:46.601368 24398 hybrid_clock.cc:648] HybridClock initialized: now 1714572526601359 us; error 22 us; skew 500 ppm
I20240501 14:08:46.603008 24398 webserver.cc:460] Webserver started at http://127.22.252.190:43351/ using document root <none> and password file <none>
I20240501 14:08:46.603508 24398 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:46.603566 24398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:46.603729 24398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:46.605309 24398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f39246b4f6f04e18a61462eefbf48a9a"
format_stamp: "Formatted at 2024-05-01 14:08:46 on dist-test-slave-bkzb"
I20240501 14:08:46.605747 24398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f39246b4f6f04e18a61462eefbf48a9a"
format_stamp: "Formatted at 2024-05-01 14:08:46 on dist-test-slave-bkzb"
I20240501 14:08:46.608851 24398 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240501 14:08:46.610745 24412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:46.611642 24398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:46.611747 24398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f39246b4f6f04e18a61462eefbf48a9a"
format_stamp: "Formatted at 2024-05-01 14:08:46 on dist-test-slave-bkzb"
I20240501 14:08:46.611825 24398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:46.623054 24398 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:46.623584 24398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:46.623730 24398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:46.629834 24398 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.190:46861
I20240501 14:08:46.629853 24464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.190:46861 every 8 connection(s)
I20240501 14:08:46.630686 24398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:08:46.633070 24465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:46.636602 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 24398
I20240501 14:08:46.636705 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:08:46.638172 24465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a: Bootstrap starting.
I20240501 14:08:46.640211 24465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:46.640895 24465 log.cc:824] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:46.642410 24465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a: No bootstrap required, opened a new log
I20240501 14:08:46.645393 24465 raft_consensus.cc:348] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f39246b4f6f04e18a61462eefbf48a9a" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 46861 } }
I20240501 14:08:46.645581 24465 raft_consensus.cc:374] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:46.645648 24465 raft_consensus.cc:729] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f39246b4f6f04e18a61462eefbf48a9a, State: Initialized, Role: FOLLOWER
I20240501 14:08:46.646015 24465 consensus_queue.cc:260] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f39246b4f6f04e18a61462eefbf48a9a" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 46861 } }
I20240501 14:08:46.646132 24465 raft_consensus.cc:388] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:46.646193 24465 raft_consensus.cc:482] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:46.646304 24465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:46.646946 24465 raft_consensus.cc:504] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f39246b4f6f04e18a61462eefbf48a9a" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 46861 } }
I20240501 14:08:46.647245 24465 leader_election.cc:302] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f39246b4f6f04e18a61462eefbf48a9a; no voters:
I20240501 14:08:46.647498 24465 leader_election.cc:288] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:46.647625 24470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:46.648267 24470 raft_consensus.cc:686] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a [term 1 LEADER]: Becoming Leader. State: Replica: f39246b4f6f04e18a61462eefbf48a9a, State: Running, Role: LEADER
I20240501 14:08:46.648502 24465 sys_catalog.cc:564] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:08:46.648555 24470 consensus_queue.cc:237] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f39246b4f6f04e18a61462eefbf48a9a" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 46861 } }
I20240501 14:08:46.649868 24472 sys_catalog.cc:455] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f39246b4f6f04e18a61462eefbf48a9a. Latest consensus state: current_term: 1 leader_uuid: "f39246b4f6f04e18a61462eefbf48a9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f39246b4f6f04e18a61462eefbf48a9a" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 46861 } } }
I20240501 14:08:46.649964 24472 sys_catalog.cc:458] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:46.650221 24471 sys_catalog.cc:455] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f39246b4f6f04e18a61462eefbf48a9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f39246b4f6f04e18a61462eefbf48a9a" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 46861 } } }
I20240501 14:08:46.650315 24471 sys_catalog.cc:458] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:46.650451 24481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:08:46.653641 24481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:08:46.658385 24481 catalog_manager.cc:1347] Generated new cluster ID: 5d20561602304ebcbce338cd19aab254
I20240501 14:08:46.658447 24481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:08:46.666584 24481 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:08:46.667641 24481 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:08:46.679107 24481 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f39246b4f6f04e18a61462eefbf48a9a: Generated new TSK 0
I20240501 14:08:46.679772 24481 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:08:46.682801 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.129:0
--local_ip_for_outbound_sockets=127.22.252.129
--webserver_interface=127.22.252.129
--webserver_port=0
--tserver_master_addrs=127.22.252.190:46861
--builtin_ntp_servers=127.22.252.148:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:08:46.783908 24489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:46.784127 24489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:46.784170 24489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:46.787147 24489 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:08:46.787211 24489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:46.787287 24489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.129
I20240501 14:08:46.790769 24489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.252.129
--webserver_port=0
--tserver_master_addrs=127.22.252.190:46861
--never_fsync=true
--heap_profile_path=/tmp/kudu.24489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.129
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:46.791613 24489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:46.792697 24489 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:46.800683 24494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:46.801386 24497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:46.801919 24495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:46.802544 24489 server_base.cc:1034] running on GCE node
I20240501 14:08:46.802920 24489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:46.803504 24489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:46.804670 24489 hybrid_clock.cc:648] HybridClock initialized: now 1714572526804650 us; error 30 us; skew 500 ppm
I20240501 14:08:46.806725 24489 webserver.cc:460] Webserver started at http://127.22.252.129:39027/ using document root <none> and password file <none>
I20240501 14:08:46.807365 24489 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:46.807433 24489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:46.807619 24489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:46.809686 24489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b27daaadc84549edb45205ee1ee6a864"
format_stamp: "Formatted at 2024-05-01 14:08:46 on dist-test-slave-bkzb"
I20240501 14:08:46.810122 24489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b27daaadc84549edb45205ee1ee6a864"
format_stamp: "Formatted at 2024-05-01 14:08:46 on dist-test-slave-bkzb"
I20240501 14:08:46.813323 24489 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:08:46.817397 24503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:46.818364 24489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:46.818464 24489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b27daaadc84549edb45205ee1ee6a864"
format_stamp: "Formatted at 2024-05-01 14:08:46 on dist-test-slave-bkzb"
I20240501 14:08:46.818545 24489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:46.835362 24489 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:46.835889 24489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:46.836059 24489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:46.836580 24489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:46.837458 24489 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:46.837505 24489 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:46.837560 24489 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:46.837589 24489 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:46.845362 24489 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.129:43809
I20240501 14:08:46.845394 24616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.129:43809 every 8 connection(s)
I20240501 14:08:46.846225 24489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:08:46.848078 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 24489
I20240501 14:08:46.848196 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:08:46.850350 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.130:0
--local_ip_for_outbound_sockets=127.22.252.130
--webserver_interface=127.22.252.130
--webserver_port=0
--tserver_master_addrs=127.22.252.190:46861
--builtin_ntp_servers=127.22.252.148:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:08:46.855190 24617 heartbeater.cc:347] Connected to a master server at 127.22.252.190:46861
I20240501 14:08:46.855393 24617 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:46.855788 24617 heartbeater.cc:511] Master 127.22.252.190:46861 requested a full tablet report, sending...
I20240501 14:08:46.857079 24429 ts_manager.cc:194] Registered new tserver with Master: b27daaadc84549edb45205ee1ee6a864 (127.22.252.129:43809)
I20240501 14:08:46.858198 24429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.129:36969
W20240501 14:08:46.952419 24620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:46.952607 24620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:46.952673 24620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:46.955370 24620 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:08:46.955430 24620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:46.955516 24620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.130
I20240501 14:08:46.958830 24620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.252.130
--webserver_port=0
--tserver_master_addrs=127.22.252.190:46861
--never_fsync=true
--heap_profile_path=/tmp/kudu.24620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.130
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:46.959642 24620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:46.960820 24620 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:46.968863 24625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:46.968935 24628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:46.969792 24626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:46.970440 24620 server_base.cc:1034] running on GCE node
I20240501 14:08:46.970782 24620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:46.971269 24620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:46.972412 24620 hybrid_clock.cc:648] HybridClock initialized: now 1714572526972393 us; error 35 us; skew 500 ppm
I20240501 14:08:46.974145 24620 webserver.cc:460] Webserver started at http://127.22.252.130:43423/ using document root <none> and password file <none>
I20240501 14:08:46.974653 24620 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:46.974716 24620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:46.974902 24620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:46.976456 24620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d32c329e43464e2eb0c7dcb39a5a8132"
format_stamp: "Formatted at 2024-05-01 14:08:46 on dist-test-slave-bkzb"
I20240501 14:08:46.976898 24620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d32c329e43464e2eb0c7dcb39a5a8132"
format_stamp: "Formatted at 2024-05-01 14:08:46 on dist-test-slave-bkzb"
I20240501 14:08:46.979976 24620 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:08:46.982024 24634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:46.982928 24620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:46.983033 24620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d32c329e43464e2eb0c7dcb39a5a8132"
format_stamp: "Formatted at 2024-05-01 14:08:46 on dist-test-slave-bkzb"
I20240501 14:08:46.983105 24620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:47.003688 24620 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:47.004261 24620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:47.004441 24620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:47.004930 24620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:47.005760 24620 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:47.005810 24620 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:47.005870 24620 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:47.005897 24620 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:47.013937 24620 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.130:46581
I20240501 14:08:47.013963 24747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.130:46581 every 8 connection(s)
I20240501 14:08:47.014776 24620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:08:47.016144 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 24620
I20240501 14:08:47.016283 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:08:47.018617 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.131:0
--local_ip_for_outbound_sockets=127.22.252.131
--webserver_interface=127.22.252.131
--webserver_port=0
--tserver_master_addrs=127.22.252.190:46861
--builtin_ntp_servers=127.22.252.148:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:08:47.024170 24748 heartbeater.cc:347] Connected to a master server at 127.22.252.190:46861
I20240501 14:08:47.024394 24748 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:47.024793 24748 heartbeater.cc:511] Master 127.22.252.190:46861 requested a full tablet report, sending...
I20240501 14:08:47.025820 24429 ts_manager.cc:194] Registered new tserver with Master: d32c329e43464e2eb0c7dcb39a5a8132 (127.22.252.130:46581)
I20240501 14:08:47.026670 24429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.130:39295
W20240501 14:08:47.121245 24751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:47.121423 24751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:47.121460 24751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:47.124339 24751 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:08:47.124387 24751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:47.124444 24751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.131
I20240501 14:08:47.127707 24751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.252.131
--webserver_port=0
--tserver_master_addrs=127.22.252.190:46861
--never_fsync=true
--heap_profile_path=/tmp/kudu.24751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.131
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:47.128480 24751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:47.129484 24751 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:47.137073 24757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:47.137835 24759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:47.138551 24756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:47.139299 24751 server_base.cc:1034] running on GCE node
I20240501 14:08:47.139652 24751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:47.140157 24751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:47.141300 24751 hybrid_clock.cc:648] HybridClock initialized: now 1714572527141289 us; error 25 us; skew 500 ppm
I20240501 14:08:47.143115 24751 webserver.cc:460] Webserver started at http://127.22.252.131:34363/ using document root <none> and password file <none>
I20240501 14:08:47.143638 24751 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:47.143699 24751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:47.143862 24751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:47.145495 24751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "1e2653ab6cfc4539ab993fe9070d5a46"
format_stamp: "Formatted at 2024-05-01 14:08:47 on dist-test-slave-bkzb"
I20240501 14:08:47.145939 24751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "1e2653ab6cfc4539ab993fe9070d5a46"
format_stamp: "Formatted at 2024-05-01 14:08:47 on dist-test-slave-bkzb"
I20240501 14:08:47.149022 24751 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:08:47.151202 24765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:47.152101 24751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:47.152201 24751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1e2653ab6cfc4539ab993fe9070d5a46"
format_stamp: "Formatted at 2024-05-01 14:08:47 on dist-test-slave-bkzb"
I20240501 14:08:47.152313 24751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:47.169940 24751 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:47.170444 24751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:47.170586 24751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:47.171051 24751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:47.171823 24751 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:47.171869 24751 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:47.171926 24751 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:47.171950 24751 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:47.180024 24751 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.131:35769
I20240501 14:08:47.180050 24878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.131:35769 every 8 connection(s)
I20240501 14:08:47.180873 24751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:08:47.184775 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 24751
I20240501 14:08:47.184852 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:08:47.186718 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.132:0
--local_ip_for_outbound_sockets=127.22.252.132
--webserver_interface=127.22.252.132
--webserver_port=0
--tserver_master_addrs=127.22.252.190:46861
--builtin_ntp_servers=127.22.252.148:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:08:47.189075 24879 heartbeater.cc:347] Connected to a master server at 127.22.252.190:46861
I20240501 14:08:47.189263 24879 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:47.189638 24879 heartbeater.cc:511] Master 127.22.252.190:46861 requested a full tablet report, sending...
I20240501 14:08:47.190534 24429 ts_manager.cc:194] Registered new tserver with Master: 1e2653ab6cfc4539ab993fe9070d5a46 (127.22.252.131:35769)
I20240501 14:08:47.191028 24429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.131:42873
W20240501 14:08:47.288558 24882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:47.288762 24882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:47.288832 24882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:47.291715 24882 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:08:47.291779 24882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:47.291867 24882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.132
I20240501 14:08:47.295322 24882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.252.132
--webserver_port=0
--tserver_master_addrs=127.22.252.190:46861
--never_fsync=true
--heap_profile_path=/tmp/kudu.24882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.132
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:47.296125 24882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:47.297187 24882 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:47.304879 24890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:47.305527 24887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:47.306249 24888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:47.306875 24882 server_base.cc:1034] running on GCE node
I20240501 14:08:47.307282 24882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:47.307804 24882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:47.308944 24882 hybrid_clock.cc:648] HybridClock initialized: now 1714572527308928 us; error 27 us; skew 500 ppm
I20240501 14:08:47.310762 24882 webserver.cc:460] Webserver started at http://127.22.252.132:46257/ using document root <none> and password file <none>
I20240501 14:08:47.311311 24882 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:47.311362 24882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:47.311534 24882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:47.313105 24882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4f74674218b34a69beff343d2da76a29"
format_stamp: "Formatted at 2024-05-01 14:08:47 on dist-test-slave-bkzb"
I20240501 14:08:47.313562 24882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4f74674218b34a69beff343d2da76a29"
format_stamp: "Formatted at 2024-05-01 14:08:47 on dist-test-slave-bkzb"
I20240501 14:08:47.316771 24882 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:08:47.318790 24896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:47.319733 24882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:47.319830 24882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4f74674218b34a69beff343d2da76a29"
format_stamp: "Formatted at 2024-05-01 14:08:47 on dist-test-slave-bkzb"
I20240501 14:08:47.319916 24882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:47.340914 24882 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:47.341509 24882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:47.341679 24882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:47.342232 24882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:47.343116 24882 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:47.343161 24882 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:47.343207 24882 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:47.343241 24882 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:47.351800 24882 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.132:33881
I20240501 14:08:47.351825 25009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.132:33881 every 8 connection(s)
I20240501 14:08:47.352747 24882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:08:47.361074 25010 heartbeater.cc:347] Connected to a master server at 127.22.252.190:46861
I20240501 14:08:47.361268 25010 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:47.361654 25010 heartbeater.cc:511] Master 127.22.252.190:46861 requested a full tablet report, sending...
I20240501 14:08:47.362257 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 24882
I20240501 14:08:47.362362 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:08:47.362579 24429 ts_manager.cc:194] Registered new tserver with Master: 4f74674218b34a69beff343d2da76a29 (127.22.252.132:33881)
I20240501 14:08:47.363143 24429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.132:41403
I20240501 14:08:47.364523 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.133:0
--local_ip_for_outbound_sockets=127.22.252.133
--webserver_interface=127.22.252.133
--webserver_port=0
--tserver_master_addrs=127.22.252.190:46861
--builtin_ntp_servers=127.22.252.148:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:08:47.465860 25013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:47.466071 25013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:47.466112 25013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:47.468936 25013 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:08:47.468995 25013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:47.469055 25013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.133
I20240501 14:08:47.472357 25013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.252.133
--webserver_port=0
--tserver_master_addrs=127.22.252.190:46861
--never_fsync=true
--heap_profile_path=/tmp/kudu.25013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.133
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:47.473114 25013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:47.474138 25013 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:47.482095 25021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:47.482290 25019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:47.482959 25018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:47.483628 25013 server_base.cc:1034] running on GCE node
I20240501 14:08:47.483968 25013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:47.484511 25013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:47.485658 25013 hybrid_clock.cc:648] HybridClock initialized: now 1714572527485634 us; error 39 us; skew 500 ppm
I20240501 14:08:47.487532 25013 webserver.cc:460] Webserver started at http://127.22.252.133:41745/ using document root <none> and password file <none>
I20240501 14:08:47.488052 25013 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:47.488113 25013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:47.488327 25013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:47.489872 25013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "1aaa098e95464df0bde4ac33ecc40095"
format_stamp: "Formatted at 2024-05-01 14:08:47 on dist-test-slave-bkzb"
I20240501 14:08:47.490315 25013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "1aaa098e95464df0bde4ac33ecc40095"
format_stamp: "Formatted at 2024-05-01 14:08:47 on dist-test-slave-bkzb"
I20240501 14:08:47.493470 25013 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:08:47.495536 25027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:47.496549 25013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:08:47.496654 25013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1aaa098e95464df0bde4ac33ecc40095"
format_stamp: "Formatted at 2024-05-01 14:08:47 on dist-test-slave-bkzb"
I20240501 14:08:47.496737 25013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:47.511134 25013 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:47.511695 25013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:47.511848 25013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:47.512403 25013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:47.513252 25013 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:47.513300 25013 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:47.513360 25013 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:47.513388 25013 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:47.521669 25013 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.133:36631
I20240501 14:08:47.521762 25140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.133:36631 every 8 connection(s)
I20240501 14:08:47.522564 25013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:08:47.529434 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 25013
I20240501 14:08:47.529553 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:08:47.530894 25141 heartbeater.cc:347] Connected to a master server at 127.22.252.190:46861
I20240501 14:08:47.531098 25141 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:47.531574 25141 heartbeater.cc:511] Master 127.22.252.190:46861 requested a full tablet report, sending...
I20240501 14:08:47.532441 24429 ts_manager.cc:194] Registered new tserver with Master: 1aaa098e95464df0bde4ac33ecc40095 (127.22.252.133:36631)
I20240501 14:08:47.532953 24429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.133:52807
I20240501 14:08:47.543469 23538 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 14:08:47.551785 24429 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47866:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 14:08:47.568706 24944 tablet_service.cc:1467] Processing CreateTablet for tablet e380fd4d283f4a85ba9f4ea3da5c3117 (DEFAULT_TABLE table=TestTable [id=25b213a07d5b4bc99bc1ad07bf8acf19]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:47.569269 25075 tablet_service.cc:1467] Processing CreateTablet for tablet e380fd4d283f4a85ba9f4ea3da5c3117 (DEFAULT_TABLE table=TestTable [id=25b213a07d5b4bc99bc1ad07bf8acf19]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:47.569272 24551 tablet_service.cc:1467] Processing CreateTablet for tablet e380fd4d283f4a85ba9f4ea3da5c3117 (DEFAULT_TABLE table=TestTable [id=25b213a07d5b4bc99bc1ad07bf8acf19]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:47.569656 24944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e380fd4d283f4a85ba9f4ea3da5c3117. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:47.570135 25075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e380fd4d283f4a85ba9f4ea3da5c3117. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:47.570134 24551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e380fd4d283f4a85ba9f4ea3da5c3117. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:47.574741 25158 tablet_bootstrap.cc:492] T e380fd4d283f4a85ba9f4ea3da5c3117 P 4f74674218b34a69beff343d2da76a29: Bootstrap starting.
I20240501 14:08:47.576328 25159 tablet_bootstrap.cc:492] T e380fd4d283f4a85ba9f4ea3da5c3117 P 1aaa098e95464df0bde4ac33ecc40095: Bootstrap starting.
I20240501 14:08:47.576325 25160 tablet_bootstrap.cc:492] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864: Bootstrap starting.
I20240501 14:08:47.576961 25158 tablet_bootstrap.cc:654] T e380fd4d283f4a85ba9f4ea3da5c3117 P 4f74674218b34a69beff343d2da76a29: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:47.577651 25158 log.cc:824] T e380fd4d283f4a85ba9f4ea3da5c3117 P 4f74674218b34a69beff343d2da76a29: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:47.578287 25159 tablet_bootstrap.cc:654] T e380fd4d283f4a85ba9f4ea3da5c3117 P 1aaa098e95464df0bde4ac33ecc40095: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:47.578287 25160 tablet_bootstrap.cc:654] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:47.579067 25159 log.cc:824] T e380fd4d283f4a85ba9f4ea3da5c3117 P 1aaa098e95464df0bde4ac33ecc40095: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:47.579067 25160 log.cc:824] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:47.579757 25158 tablet_bootstrap.cc:492] T e380fd4d283f4a85ba9f4ea3da5c3117 P 4f74674218b34a69beff343d2da76a29: No bootstrap required, opened a new log
I20240501 14:08:47.579919 25158 ts_tablet_manager.cc:1367] T e380fd4d283f4a85ba9f4ea3da5c3117 P 4f74674218b34a69beff343d2da76a29: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:08:47.580714 25160 tablet_bootstrap.cc:492] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864: No bootstrap required, opened a new log
I20240501 14:08:47.580714 25159 tablet_bootstrap.cc:492] T e380fd4d283f4a85ba9f4ea3da5c3117 P 1aaa098e95464df0bde4ac33ecc40095: No bootstrap required, opened a new log
I20240501 14:08:47.580881 25160 ts_tablet_manager.cc:1367] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:08:47.580881 25159 ts_tablet_manager.cc:1367] T e380fd4d283f4a85ba9f4ea3da5c3117 P 1aaa098e95464df0bde4ac33ecc40095: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240501 14:08:47.582751 25158 raft_consensus.cc:348] T e380fd4d283f4a85ba9f4ea3da5c3117 P 4f74674218b34a69beff343d2da76a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f74674218b34a69beff343d2da76a29" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 33881 } } peers { permanent_uuid: "1aaa098e95464df0bde4ac33ecc40095" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 36631 } } peers { permanent_uuid: "b27daaadc84549edb45205ee1ee6a864" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 43809 } }
I20240501 14:08:47.582926 25158 raft_consensus.cc:374] T e380fd4d283f4a85ba9f4ea3da5c3117 P 4f74674218b34a69beff343d2da76a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:47.582981 25158 raft_consensus.cc:729] T e380fd4d283f4a85ba9f4ea3da5c3117 P 4f74674218b34a69beff343d2da76a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f74674218b34a69beff343d2da76a29, State: Initialized, Role: FOLLOWER
I20240501 14:08:47.583379 25158 consensus_queue.cc:260] T e380fd4d283f4a85ba9f4ea3da5c3117 P 4f74674218b34a69beff343d2da76a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f74674218b34a69beff343d2da76a29" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 33881 } } peers { permanent_uuid: "1aaa098e95464df0bde4ac33ecc40095" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 36631 } } peers { permanent_uuid: "b27daaadc84549edb45205ee1ee6a864" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 43809 } }
I20240501 14:08:47.584250 25158 ts_tablet_manager.cc:1398] T e380fd4d283f4a85ba9f4ea3da5c3117 P 4f74674218b34a69beff343d2da76a29: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 14:08:47.584625 25010 heartbeater.cc:503] Master 127.22.252.190:46861 was elected leader, sending a full tablet report...
I20240501 14:08:47.584515 25160 raft_consensus.cc:348] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f74674218b34a69beff343d2da76a29" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 33881 } } peers { permanent_uuid: "1aaa098e95464df0bde4ac33ecc40095" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 36631 } } peers { permanent_uuid: "b27daaadc84549edb45205ee1ee6a864" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 43809 } }
I20240501 14:08:47.584515 25159 raft_consensus.cc:348] T e380fd4d283f4a85ba9f4ea3da5c3117 P 1aaa098e95464df0bde4ac33ecc40095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f74674218b34a69beff343d2da76a29" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 33881 } } peers { permanent_uuid: "1aaa098e95464df0bde4ac33ecc40095" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 36631 } } peers { permanent_uuid: "b27daaadc84549edb45205ee1ee6a864" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 43809 } }
I20240501 14:08:47.584689 25159 raft_consensus.cc:374] T e380fd4d283f4a85ba9f4ea3da5c3117 P 1aaa098e95464df0bde4ac33ecc40095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:47.584689 25160 raft_consensus.cc:374] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:47.584751 25160 raft_consensus.cc:729] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b27daaadc84549edb45205ee1ee6a864, State: Initialized, Role: FOLLOWER
I20240501 14:08:47.584751 25159 raft_consensus.cc:729] T e380fd4d283f4a85ba9f4ea3da5c3117 P 1aaa098e95464df0bde4ac33ecc40095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aaa098e95464df0bde4ac33ecc40095, State: Initialized, Role: FOLLOWER
I20240501 14:08:47.585204 25160 consensus_queue.cc:260] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f74674218b34a69beff343d2da76a29" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 33881 } } peers { permanent_uuid: "1aaa098e95464df0bde4ac33ecc40095" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 36631 } } peers { permanent_uuid: "b27daaadc84549edb45205ee1ee6a864" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 43809 } }
I20240501 14:08:47.585204 25159 consensus_queue.cc:260] T e380fd4d283f4a85ba9f4ea3da5c3117 P 1aaa098e95464df0bde4ac33ecc40095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f74674218b34a69beff343d2da76a29" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 33881 } } peers { permanent_uuid: "1aaa098e95464df0bde4ac33ecc40095" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 36631 } } peers { permanent_uuid: "b27daaadc84549edb45205ee1ee6a864" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 43809 } }
I20240501 14:08:47.586001 25159 ts_tablet_manager.cc:1398] T e380fd4d283f4a85ba9f4ea3da5c3117 P 1aaa098e95464df0bde4ac33ecc40095: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20240501 14:08:47.585991 25160 ts_tablet_manager.cc:1398] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 14:08:47.586350 25141 heartbeater.cc:503] Master 127.22.252.190:46861 was elected leader, sending a full tablet report...
I20240501 14:08:47.586553 24617 heartbeater.cc:503] Master 127.22.252.190:46861 was elected leader, sending a full tablet report...
W20240501 14:08:47.597438 24618 tablet.cc:2369] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:08:47.603547 25011 tablet.cc:2369] T e380fd4d283f4a85ba9f4ea3da5c3117 P 4f74674218b34a69beff343d2da76a29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:08:47.755156 25165 raft_consensus.cc:482] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:47.755362 25165 raft_consensus.cc:504] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f74674218b34a69beff343d2da76a29" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 33881 } } peers { permanent_uuid: "1aaa098e95464df0bde4ac33ecc40095" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 36631 } } peers { permanent_uuid: "b27daaadc84549edb45205ee1ee6a864" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 43809 } }
I20240501 14:08:47.756541 25165 leader_election.cc:288] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f74674218b34a69beff343d2da76a29 (127.22.252.132:33881), 1aaa098e95464df0bde4ac33ecc40095 (127.22.252.133:36631)
I20240501 14:08:47.760210 25095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e380fd4d283f4a85ba9f4ea3da5c3117" candidate_uuid: "b27daaadc84549edb45205ee1ee6a864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aaa098e95464df0bde4ac33ecc40095" is_pre_election: true
I20240501 14:08:47.760212 24964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e380fd4d283f4a85ba9f4ea3da5c3117" candidate_uuid: "b27daaadc84549edb45205ee1ee6a864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f74674218b34a69beff343d2da76a29" is_pre_election: true
I20240501 14:08:47.760511 25095 raft_consensus.cc:2454] T e380fd4d283f4a85ba9f4ea3da5c3117 P 1aaa098e95464df0bde4ac33ecc40095 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b27daaadc84549edb45205ee1ee6a864 in term 0.
I20240501 14:08:47.760516 24964 raft_consensus.cc:2454] T e380fd4d283f4a85ba9f4ea3da5c3117 P 4f74674218b34a69beff343d2da76a29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b27daaadc84549edb45205ee1ee6a864 in term 0.
I20240501 14:08:47.760942 24506 leader_election.cc:302] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aaa098e95464df0bde4ac33ecc40095, b27daaadc84549edb45205ee1ee6a864; no voters:
I20240501 14:08:47.761165 25165 raft_consensus.cc:2781] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:47.761243 25165 raft_consensus.cc:482] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:47.761296 25165 raft_consensus.cc:3037] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:47.762140 25165 raft_consensus.cc:504] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f74674218b34a69beff343d2da76a29" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 33881 } } peers { permanent_uuid: "1aaa098e95464df0bde4ac33ecc40095" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 36631 } } peers { permanent_uuid: "b27daaadc84549edb45205ee1ee6a864" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 43809 } }
I20240501 14:08:47.762490 25165 leader_election.cc:288] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864 [CANDIDATE]: Term 1 election: Requested vote from peers 4f74674218b34a69beff343d2da76a29 (127.22.252.132:33881), 1aaa098e95464df0bde4ac33ecc40095 (127.22.252.133:36631)
I20240501 14:08:47.762802 24964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e380fd4d283f4a85ba9f4ea3da5c3117" candidate_uuid: "b27daaadc84549edb45205ee1ee6a864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f74674218b34a69beff343d2da76a29"
I20240501 14:08:47.762929 24964 raft_consensus.cc:3037] T e380fd4d283f4a85ba9f4ea3da5c3117 P 4f74674218b34a69beff343d2da76a29 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:47.763813 24964 raft_consensus.cc:2454] T e380fd4d283f4a85ba9f4ea3da5c3117 P 4f74674218b34a69beff343d2da76a29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b27daaadc84549edb45205ee1ee6a864 in term 1.
I20240501 14:08:47.764076 25095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e380fd4d283f4a85ba9f4ea3da5c3117" candidate_uuid: "b27daaadc84549edb45205ee1ee6a864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aaa098e95464df0bde4ac33ecc40095"
I20240501 14:08:47.764145 24504 leader_election.cc:302] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f74674218b34a69beff343d2da76a29, b27daaadc84549edb45205ee1ee6a864; no voters:
I20240501 14:08:47.764214 25095 raft_consensus.cc:3037] T e380fd4d283f4a85ba9f4ea3da5c3117 P 1aaa098e95464df0bde4ac33ecc40095 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:47.764329 25165 raft_consensus.cc:2781] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:47.764564 25165 raft_consensus.cc:686] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864 [term 1 LEADER]: Becoming Leader. State: Replica: b27daaadc84549edb45205ee1ee6a864, State: Running, Role: LEADER
I20240501 14:08:47.764891 25165 consensus_queue.cc:237] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f74674218b34a69beff343d2da76a29" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 33881 } } peers { permanent_uuid: "1aaa098e95464df0bde4ac33ecc40095" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 36631 } } peers { permanent_uuid: "b27daaadc84549edb45205ee1ee6a864" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 43809 } }
I20240501 14:08:47.765273 25095 raft_consensus.cc:2454] T e380fd4d283f4a85ba9f4ea3da5c3117 P 1aaa098e95464df0bde4ac33ecc40095 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b27daaadc84549edb45205ee1ee6a864 in term 1.
I20240501 14:08:47.767875 24428 catalog_manager.cc:5484] T e380fd4d283f4a85ba9f4ea3da5c3117 P b27daaadc84549edb45205ee1ee6a864 reported cstate change: term changed from 0 to 1, leader changed from <none> to b27daaadc84549edb45205ee1ee6a864 (127.22.252.129). New cstate: current_term: 1 leader_uuid: "b27daaadc84549edb45205ee1ee6a864" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f74674218b34a69beff343d2da76a29" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 33881 } } peers { permanent_uuid: "1aaa098e95464df0bde4ac33ecc40095" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 36631 } } peers { permanent_uuid: "b27daaadc84549edb45205ee1ee6a864" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 43809 } } }
W20240501 14:08:47.773449 25142 tablet.cc:2369] T e380fd4d283f4a85ba9f4ea3da5c3117 P 1aaa098e95464df0bde4ac33ecc40095: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:08:47.797101 23538 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 14:08:47.798498 23538 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b27daaadc84549edb45205ee1ee6a864 to finish bootstrapping
I20240501 14:08:47.806941 23538 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d32c329e43464e2eb0c7dcb39a5a8132 to finish bootstrapping
I20240501 14:08:47.812601 23538 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 1e2653ab6cfc4539ab993fe9070d5a46 to finish bootstrapping
I20240501 14:08:47.818360 23538 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4f74674218b34a69beff343d2da76a29 to finish bootstrapping
I20240501 14:08:47.822994 23538 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1aaa098e95464df0bde4ac33ecc40095 to finish bootstrapping
I20240501 14:08:47.827690 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 24489
I20240501 14:08:48.019030 24944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.020354 24682 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.021462 24813 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.022027 25075 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.028410 24748 heartbeater.cc:503] Master 127.22.252.190:46861 was elected leader, sending a full tablet report...
I20240501 14:08:48.091931 24944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.092109 25075 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.092765 24813 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.093986 24682 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.192843 24879 heartbeater.cc:503] Master 127.22.252.190:46861 was elected leader, sending a full tablet report...
I20240501 14:08:48.193902 25075 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.194679 24944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.195086 24813 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.197455 24682 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.402990 25075 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.403259 24944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.404495 24813 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.404623 24682 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.474833 24813 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.474870 25075 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.475021 24682 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.475862 24944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.571909 25075 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.573741 24682 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.573984 24944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.574368 24813 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:48.608088 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 24620
I20240501 14:08:48.613535 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 24751
I20240501 14:08:48.618948 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 24882
I20240501 14:08:48.624485 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 25013
I20240501 14:08:48.629897 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 24398
2024-05-01T14:08:48Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/0 (2177 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2177 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/0
I20240501 14:08:48.651993 23538 test_util.cc:268] Using random seed: 109252309
I20240501 14:08:48.652858 23538 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:08:48.652899 23538 ts_itest-base.cc:113] --------------
I20240501 14:08:48.652909 23538 ts_itest-base.cc:114] 7 tablet servers
I20240501 14:08:48.652920 23538 ts_itest-base.cc:115] 1 replicas per TS
I20240501 14:08:48.652930 23538 ts_itest-base.cc:116] --------------
2024-05-01T14:08:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:08:48Z Disabled control of system clock
I20240501 14:08:48.660660 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.252.190:44241
--webserver_interface=127.22.252.190
--webserver_port=0
--builtin_ntp_servers=127.22.252.148:35091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.252.190:44241
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 14:08:48.760996 25356 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 14:08:48.761180 25356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:48.761204 25356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:48.761258 25356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:48.763926 25356 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:08:48.763999 25356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:08:48.764020 25356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:48.764037 25356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:08:48.764053 25356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:08:48.767293 25356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:35091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.252.190:44241
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.252.190:44241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.252.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.25356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:48.768077 25356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:48.769084 25356 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:48.775949 25364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:48.776630 25362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:48.776824 25361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:48.777477 25356 server_base.cc:1034] running on GCE node
I20240501 14:08:48.777904 25356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:48.778710 25356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:48.779858 25356 hybrid_clock.cc:648] HybridClock initialized: now 1714572528779842 us; error 31 us; skew 500 ppm
I20240501 14:08:48.781620 25356 webserver.cc:460] Webserver started at http://127.22.252.190:45651/ using document root <none> and password file <none>
I20240501 14:08:48.782124 25356 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:48.782181 25356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:48.782362 25356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:48.783892 25356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ee2720787e344e1a83ff5dc97e1e323e"
format_stamp: "Formatted at 2024-05-01 14:08:48 on dist-test-slave-bkzb"
I20240501 14:08:48.784396 25356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ee2720787e344e1a83ff5dc97e1e323e"
format_stamp: "Formatted at 2024-05-01 14:08:48 on dist-test-slave-bkzb"
I20240501 14:08:48.787556 25356 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:48.789556 25370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:48.790455 25356 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:48.790555 25356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ee2720787e344e1a83ff5dc97e1e323e"
format_stamp: "Formatted at 2024-05-01 14:08:48 on dist-test-slave-bkzb"
I20240501 14:08:48.790628 25356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:48.802660 25356 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:48.803169 25356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:48.803308 25356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:48.809489 25356 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.190:44241
I20240501 14:08:48.809500 25422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.190:44241 every 8 connection(s)
I20240501 14:08:48.810364 25356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:08:48.812667 25423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:48.815371 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 25356
I20240501 14:08:48.815483 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:08:48.818022 25423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e: Bootstrap starting.
I20240501 14:08:48.819882 25423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:48.820549 25423 log.cc:824] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:48.822469 25423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e: No bootstrap required, opened a new log
I20240501 14:08:48.825074 25423 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2720787e344e1a83ff5dc97e1e323e" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 44241 } }
I20240501 14:08:48.825256 25423 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:48.825296 25423 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee2720787e344e1a83ff5dc97e1e323e, State: Initialized, Role: FOLLOWER
I20240501 14:08:48.825639 25423 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2720787e344e1a83ff5dc97e1e323e" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 44241 } }
I20240501 14:08:48.825747 25423 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:48.825789 25423 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:48.825863 25423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:48.826495 25423 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2720787e344e1a83ff5dc97e1e323e" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 44241 } }
I20240501 14:08:48.826762 25423 leader_election.cc:302] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee2720787e344e1a83ff5dc97e1e323e; no voters:
I20240501 14:08:48.827009 25423 leader_election.cc:288] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:48.827100 25428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:48.827646 25428 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e [term 1 LEADER]: Becoming Leader. State: Replica: ee2720787e344e1a83ff5dc97e1e323e, State: Running, Role: LEADER
I20240501 14:08:48.827893 25428 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2720787e344e1a83ff5dc97e1e323e" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 44241 } }
I20240501 14:08:48.827942 25423 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:08:48.829258 25430 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee2720787e344e1a83ff5dc97e1e323e. Latest consensus state: current_term: 1 leader_uuid: "ee2720787e344e1a83ff5dc97e1e323e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2720787e344e1a83ff5dc97e1e323e" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 44241 } } }
I20240501 14:08:48.829373 25430 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:48.829680 25429 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee2720787e344e1a83ff5dc97e1e323e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2720787e344e1a83ff5dc97e1e323e" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 44241 } } }
I20240501 14:08:48.829720 25437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:08:48.829790 25429 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:48.832969 25437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:08:48.838044 25437 catalog_manager.cc:1347] Generated new cluster ID: 509e11b9818c4a0c87d056c48d652bbd
I20240501 14:08:48.838099 25437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:08:48.843050 25437 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:08:48.843835 25437 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:08:48.851867 25437 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ee2720787e344e1a83ff5dc97e1e323e: Generated new TSK 0
I20240501 14:08:48.852643 25437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:08:48.857439 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.129:0
--local_ip_for_outbound_sockets=127.22.252.129
--webserver_interface=127.22.252.129
--webserver_port=0
--tserver_master_addrs=127.22.252.190:44241
--builtin_ntp_servers=127.22.252.148:35091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:08:48.980412 25447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:48.980597 25447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:48.980640 25447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:48.983527 25447 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:08:48.983577 25447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:48.983652 25447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.129
I20240501 14:08:48.987071 25447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:35091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.252.129
--webserver_port=0
--tserver_master_addrs=127.22.252.190:44241
--never_fsync=true
--heap_profile_path=/tmp/kudu.25447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.129
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:48.987851 25447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:48.988916 25447 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:48.997022 25452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:48.997792 25455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:48.998386 25453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:48.998981 25447 server_base.cc:1034] running on GCE node
I20240501 14:08:48.999320 25447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:48.999826 25447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:49.000973 25447 hybrid_clock.cc:648] HybridClock initialized: now 1714572529000959 us; error 40 us; skew 500 ppm
I20240501 14:08:49.002830 25447 webserver.cc:460] Webserver started at http://127.22.252.129:46733/ using document root <none> and password file <none>
I20240501 14:08:49.003384 25447 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:49.003445 25447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:49.003618 25447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:49.005342 25447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e8f33be94f864561bec2e185b5a695c3"
format_stamp: "Formatted at 2024-05-01 14:08:48 on dist-test-slave-bkzb"
I20240501 14:08:49.005793 25447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e8f33be94f864561bec2e185b5a695c3"
format_stamp: "Formatted at 2024-05-01 14:08:48 on dist-test-slave-bkzb"
I20240501 14:08:49.009068 25447 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:08:49.011269 25461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.012298 25447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:49.012400 25447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e8f33be94f864561bec2e185b5a695c3"
format_stamp: "Formatted at 2024-05-01 14:08:48 on dist-test-slave-bkzb"
I20240501 14:08:49.012485 25447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:49.026580 25447 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:49.027166 25447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:49.027323 25447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:49.027875 25447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:49.028827 25447 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:49.028878 25447 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.028935 25447 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:49.029022 25447 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.037235 25447 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.129:44353
I20240501 14:08:49.037281 25574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.129:44353 every 8 connection(s)
I20240501 14:08:49.038112 25447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:08:49.043694 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 25447
I20240501 14:08:49.043785 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:08:49.046012 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.130:0
--local_ip_for_outbound_sockets=127.22.252.130
--webserver_interface=127.22.252.130
--webserver_port=0
--tserver_master_addrs=127.22.252.190:44241
--builtin_ntp_servers=127.22.252.148:35091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:08:49.046520 25575 heartbeater.cc:347] Connected to a master server at 127.22.252.190:44241
I20240501 14:08:49.046710 25575 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:49.047142 25575 heartbeater.cc:511] Master 127.22.252.190:44241 requested a full tablet report, sending...
I20240501 14:08:49.048568 25387 ts_manager.cc:194] Registered new tserver with Master: e8f33be94f864561bec2e185b5a695c3 (127.22.252.129:44353)
I20240501 14:08:49.049649 25387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.129:40941
W20240501 14:08:49.151813 25578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:49.152030 25578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:49.152102 25578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:49.155042 25578 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:08:49.155104 25578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:49.155192 25578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.130
I20240501 14:08:49.158555 25578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:35091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.252.130
--webserver_port=0
--tserver_master_addrs=127.22.252.190:44241
--never_fsync=true
--heap_profile_path=/tmp/kudu.25578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.130
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:49.159333 25578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:49.160346 25578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:49.167697 25584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.168412 25583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.169174 25586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:49.169795 25578 server_base.cc:1034] running on GCE node
I20240501 14:08:49.170096 25578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:49.170567 25578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:49.171700 25578 hybrid_clock.cc:648] HybridClock initialized: now 1714572529171682 us; error 32 us; skew 500 ppm
I20240501 14:08:49.173347 25578 webserver.cc:460] Webserver started at http://127.22.252.130:36441/ using document root <none> and password file <none>
I20240501 14:08:49.173903 25578 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:49.173959 25578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:49.174108 25578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:49.175635 25578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c6717727de344c0b90dd50fd5e70883d"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-bkzb"
I20240501 14:08:49.176084 25578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c6717727de344c0b90dd50fd5e70883d"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-bkzb"
I20240501 14:08:49.179116 25578 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20240501 14:08:49.181108 25592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.182089 25578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240501 14:08:49.182179 25578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c6717727de344c0b90dd50fd5e70883d"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-bkzb"
I20240501 14:08:49.182255 25578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:49.196871 25578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:49.197393 25578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:49.197513 25578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:49.197942 25578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:49.198750 25578 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:49.198796 25578 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.198832 25578 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:49.198853 25578 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.206892 25578 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.130:44715
I20240501 14:08:49.206952 25705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.130:44715 every 8 connection(s)
I20240501 14:08:49.207732 25578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:08:49.212018 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 25578
I20240501 14:08:49.212132 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:08:49.214339 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.131:0
--local_ip_for_outbound_sockets=127.22.252.131
--webserver_interface=127.22.252.131
--webserver_port=0
--tserver_master_addrs=127.22.252.190:44241
--builtin_ntp_servers=127.22.252.148:35091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:08:49.216192 25706 heartbeater.cc:347] Connected to a master server at 127.22.252.190:44241
I20240501 14:08:49.216410 25706 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:49.216838 25706 heartbeater.cc:511] Master 127.22.252.190:44241 requested a full tablet report, sending...
I20240501 14:08:49.217841 25387 ts_manager.cc:194] Registered new tserver with Master: c6717727de344c0b90dd50fd5e70883d (127.22.252.130:44715)
I20240501 14:08:49.218686 25387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.130:55957
W20240501 14:08:49.316668 25709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:49.316854 25709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:49.316895 25709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:49.319880 25709 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:08:49.319941 25709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:49.320004 25709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.131
I20240501 14:08:49.323801 25709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:35091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.252.131
--webserver_port=0
--tserver_master_addrs=127.22.252.190:44241
--never_fsync=true
--heap_profile_path=/tmp/kudu.25709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.131
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:49.324725 25709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:49.325798 25709 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:49.333561 25714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.334208 25717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.334756 25715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:49.335628 25709 server_base.cc:1034] running on GCE node
I20240501 14:08:49.335937 25709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:49.336447 25709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:49.337589 25709 hybrid_clock.cc:648] HybridClock initialized: now 1714572529337577 us; error 24 us; skew 500 ppm
I20240501 14:08:49.339341 25709 webserver.cc:460] Webserver started at http://127.22.252.131:40119/ using document root <none> and password file <none>
I20240501 14:08:49.339851 25709 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:49.339907 25709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:49.340059 25709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:49.341636 25709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8a9c183caefa4437b53752c1203b8bb7"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-bkzb"
I20240501 14:08:49.342072 25709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8a9c183caefa4437b53752c1203b8bb7"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-bkzb"
I20240501 14:08:49.345119 25709 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240501 14:08:49.347187 25723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.348446 25709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:08:49.348546 25709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8a9c183caefa4437b53752c1203b8bb7"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-bkzb"
I20240501 14:08:49.348637 25709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:49.367643 25709 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:49.368176 25709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:49.368352 25709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:49.368846 25709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:49.369645 25709 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:49.369689 25709 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.369730 25709 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:49.369762 25709 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.377862 25709 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.131:36947
I20240501 14:08:49.377909 25836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.131:36947 every 8 connection(s)
I20240501 14:08:49.378727 25709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:08:49.380064 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 25709
I20240501 14:08:49.380188 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:08:49.382283 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.132:0
--local_ip_for_outbound_sockets=127.22.252.132
--webserver_interface=127.22.252.132
--webserver_port=0
--tserver_master_addrs=127.22.252.190:44241
--builtin_ntp_servers=127.22.252.148:35091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:08:49.387732 25837 heartbeater.cc:347] Connected to a master server at 127.22.252.190:44241
I20240501 14:08:49.387924 25837 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:49.388357 25837 heartbeater.cc:511] Master 127.22.252.190:44241 requested a full tablet report, sending...
I20240501 14:08:49.389278 25387 ts_manager.cc:194] Registered new tserver with Master: 8a9c183caefa4437b53752c1203b8bb7 (127.22.252.131:36947)
I20240501 14:08:49.390074 25387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.131:53759
W20240501 14:08:49.483742 25840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:49.483928 25840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:49.483966 25840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:49.486836 25840 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:08:49.486908 25840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:49.486971 25840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.132
I20240501 14:08:49.490336 25840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:35091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.252.132
--webserver_port=0
--tserver_master_addrs=127.22.252.190:44241
--never_fsync=true
--heap_profile_path=/tmp/kudu.25840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.132
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:49.491088 25840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:49.492013 25840 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:49.499742 25846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.499910 25848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.500692 25845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:49.501343 25840 server_base.cc:1034] running on GCE node
I20240501 14:08:49.501667 25840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:49.502144 25840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:49.503301 25840 hybrid_clock.cc:648] HybridClock initialized: now 1714572529503293 us; error 26 us; skew 500 ppm
I20240501 14:08:49.504920 25840 webserver.cc:460] Webserver started at http://127.22.252.132:34309/ using document root <none> and password file <none>
I20240501 14:08:49.505424 25840 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:49.505484 25840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:49.505645 25840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:49.507256 25840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "86b991e2c8414b6698e51851d34bc8f2"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-bkzb"
I20240501 14:08:49.507710 25840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "86b991e2c8414b6698e51851d34bc8f2"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-bkzb"
I20240501 14:08:49.510902 25840 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:08:49.513002 25854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.513963 25840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:49.514089 25840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "86b991e2c8414b6698e51851d34bc8f2"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-bkzb"
I20240501 14:08:49.514178 25840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:49.526934 25840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:49.527542 25840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:49.527680 25840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:49.528196 25840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:49.529078 25840 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:49.529129 25840 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.529191 25840 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:49.529220 25840 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.537139 25840 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.132:37205
I20240501 14:08:49.537194 25967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.132:37205 every 8 connection(s)
I20240501 14:08:49.538024 25840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:08:49.546190 25968 heartbeater.cc:347] Connected to a master server at 127.22.252.190:44241
I20240501 14:08:49.546386 25968 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:49.546782 25968 heartbeater.cc:511] Master 127.22.252.190:44241 requested a full tablet report, sending...
I20240501 14:08:49.547636 25387 ts_manager.cc:194] Registered new tserver with Master: 86b991e2c8414b6698e51851d34bc8f2 (127.22.252.132:37205)
I20240501 14:08:49.547937 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 25840
I20240501 14:08:49.548058 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:08:49.548337 25387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.132:60201
I20240501 14:08:49.550042 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.133:0
--local_ip_for_outbound_sockets=127.22.252.133
--webserver_interface=127.22.252.133
--webserver_port=0
--tserver_master_addrs=127.22.252.190:44241
--builtin_ntp_servers=127.22.252.148:35091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:08:49.654242 25971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:49.654451 25971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:49.654510 25971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:49.657419 25971 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:08:49.657477 25971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:49.657580 25971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.133
I20240501 14:08:49.661039 25971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:35091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.252.133
--webserver_port=0
--tserver_master_addrs=127.22.252.190:44241
--never_fsync=true
--heap_profile_path=/tmp/kudu.25971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.133
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:49.661803 25971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:49.662813 25971 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:49.670229 25976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.671007 25977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.671576 25979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:49.672444 25971 server_base.cc:1034] running on GCE node
I20240501 14:08:49.672735 25971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:49.673205 25971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:49.674351 25971 hybrid_clock.cc:648] HybridClock initialized: now 1714572529674330 us; error 33 us; skew 500 ppm
I20240501 14:08:49.675921 25971 webserver.cc:460] Webserver started at http://127.22.252.133:35765/ using document root <none> and password file <none>
I20240501 14:08:49.676460 25971 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:49.676513 25971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:49.676678 25971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:49.678207 25971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "e9e15d9915264a62a5c9e68f67872199"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-bkzb"
I20240501 14:08:49.678673 25971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "e9e15d9915264a62a5c9e68f67872199"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-bkzb"
I20240501 14:08:49.681711 25971 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:08:49.683602 25985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.684610 25971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:08:49.684710 25971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e9e15d9915264a62a5c9e68f67872199"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-bkzb"
I20240501 14:08:49.684808 25971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:49.701390 25971 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:49.701882 25971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:49.702004 25971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:49.702410 25971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:49.703167 25971 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:49.703222 25971 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.703256 25971 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:49.703275 25971 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.711257 25971 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.133:42377
I20240501 14:08:49.711282 26098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.133:42377 every 8 connection(s)
I20240501 14:08:49.712051 25971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:08:49.715368 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 25971
I20240501 14:08:49.715497 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:08:49.717515 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.134:0
--local_ip_for_outbound_sockets=127.22.252.134
--webserver_interface=127.22.252.134
--webserver_port=0
--tserver_master_addrs=127.22.252.190:44241
--builtin_ntp_servers=127.22.252.148:35091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:08:49.720695 26099 heartbeater.cc:347] Connected to a master server at 127.22.252.190:44241
I20240501 14:08:49.720890 26099 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:49.721282 26099 heartbeater.cc:511] Master 127.22.252.190:44241 requested a full tablet report, sending...
I20240501 14:08:49.722138 25387 ts_manager.cc:194] Registered new tserver with Master: e9e15d9915264a62a5c9e68f67872199 (127.22.252.133:42377)
I20240501 14:08:49.722646 25387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.133:53809
W20240501 14:08:49.818799 26102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:49.818995 26102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:49.819047 26102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:49.821942 26102 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:08:49.821998 26102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:49.822116 26102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.134
I20240501 14:08:49.825572 26102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:35091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.252.134
--webserver_port=0
--tserver_master_addrs=127.22.252.190:44241
--never_fsync=true
--heap_profile_path=/tmp/kudu.26102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.134
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:49.826375 26102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:49.827384 26102 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:49.835204 26110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.835363 26108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:49.836035 26107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:49.836753 26102 server_base.cc:1034] running on GCE node
I20240501 14:08:49.837087 26102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:49.837576 26102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:49.838717 26102 hybrid_clock.cc:648] HybridClock initialized: now 1714572529838702 us; error 28 us; skew 500 ppm
I20240501 14:08:49.840337 26102 webserver.cc:460] Webserver started at http://127.22.252.134:43823/ using document root <none> and password file <none>
I20240501 14:08:49.840843 26102 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:49.840902 26102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:49.841079 26102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:49.842561 26102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "0e3ebe10573a4863b3734daf8f3c4382"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-bkzb"
I20240501 14:08:49.843032 26102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "0e3ebe10573a4863b3734daf8f3c4382"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-bkzb"
I20240501 14:08:49.846081 26102 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:08:49.848093 26116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.849030 26102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:49.849130 26102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "0e3ebe10573a4863b3734daf8f3c4382"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-bkzb"
I20240501 14:08:49.849210 26102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:49.876513 26102 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:49.877055 26102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:49.877199 26102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:49.877686 26102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:49.878521 26102 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:49.878569 26102 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.878626 26102 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:49.878656 26102 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:49.886400 26102 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.134:42583
I20240501 14:08:49.886437 26229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.134:42583 every 8 connection(s)
I20240501 14:08:49.887287 26102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:08:49.893198 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 26102
I20240501 14:08:49.893311 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:08:49.895059 26230 heartbeater.cc:347] Connected to a master server at 127.22.252.190:44241
I20240501 14:08:49.895263 26230 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:49.895378 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.135:0
--local_ip_for_outbound_sockets=127.22.252.135
--webserver_interface=127.22.252.135
--webserver_port=0
--tserver_master_addrs=127.22.252.190:44241
--builtin_ntp_servers=127.22.252.148:35091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:08:49.895675 26230 heartbeater.cc:511] Master 127.22.252.190:44241 requested a full tablet report, sending...
I20240501 14:08:49.896581 25386 ts_manager.cc:194] Registered new tserver with Master: 0e3ebe10573a4863b3734daf8f3c4382 (127.22.252.134:42583)
I20240501 14:08:49.897135 25386 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.134:34159
W20240501 14:08:49.997073 26233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:49.997256 26233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:49.997294 26233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:50.000149 26233 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:08:50.000201 26233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:50.000306 26233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.135
I20240501 14:08:50.003630 26233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:35091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.252.135
--webserver_port=0
--tserver_master_addrs=127.22.252.190:44241
--never_fsync=true
--heap_profile_path=/tmp/kudu.26233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.135
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:50.004436 26233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:50.005452 26233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:50.012812 26238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:50.013487 26239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:50.014176 26241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:50.014904 26233 server_base.cc:1034] running on GCE node
I20240501 14:08:50.015261 26233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:50.015777 26233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:50.016922 26233 hybrid_clock.cc:648] HybridClock initialized: now 1714572530016902 us; error 31 us; skew 500 ppm
I20240501 14:08:50.018611 26233 webserver.cc:460] Webserver started at http://127.22.252.135:40343/ using document root <none> and password file <none>
I20240501 14:08:50.019141 26233 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:50.019197 26233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:50.019366 26233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:50.021073 26233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "3df2596a705349f185baa3b69dabfc25"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-bkzb"
I20240501 14:08:50.021520 26233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "3df2596a705349f185baa3b69dabfc25"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-bkzb"
I20240501 14:08:50.024637 26233 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:08:50.026757 26247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.027688 26233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:50.027788 26233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "3df2596a705349f185baa3b69dabfc25"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-bkzb"
I20240501 14:08:50.027863 26233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:50.050974 26233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:50.051561 26233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:50.051594 25575 heartbeater.cc:503] Master 127.22.252.190:44241 was elected leader, sending a full tablet report...
I20240501 14:08:50.051720 26233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:50.052330 26233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:50.053251 26233 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:50.053311 26233 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.053355 26233 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:50.053383 26233 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.061488 26233 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.135:39631
I20240501 14:08:50.061514 26360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.135:39631 every 8 connection(s)
I20240501 14:08:50.062305 26233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:08:50.070088 26361 heartbeater.cc:347] Connected to a master server at 127.22.252.190:44241
I20240501 14:08:50.070271 26361 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:50.070495 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 26233
I20240501 14:08:50.070590 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 14:08:50.070708 26361 heartbeater.cc:511] Master 127.22.252.190:44241 requested a full tablet report, sending...
I20240501 14:08:50.071513 25387 ts_manager.cc:194] Registered new tserver with Master: 3df2596a705349f185baa3b69dabfc25 (127.22.252.135:39631)
I20240501 14:08:50.072151 25387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.135:49079
I20240501 14:08:50.072710 23538 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 14:08:50.080714 25387 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58550:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 14:08:50.094740 25902 tablet_service.cc:1467] Processing CreateTablet for tablet 844ba8b3ff6c4d88a101d1c38e7ad0b3 (DEFAULT_TABLE table=TestTable [id=930ce210d3ad438092052845d153d9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:50.095495 25902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 844ba8b3ff6c4d88a101d1c38e7ad0b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:50.100076 26375 tablet_bootstrap.cc:492] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2: Bootstrap starting.
I20240501 14:08:50.101917 26375 tablet_bootstrap.cc:654] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:50.102583 26375 log.cc:824] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:50.103945 26375 tablet_bootstrap.cc:492] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2: No bootstrap required, opened a new log
I20240501 14:08:50.104091 26375 ts_tablet_manager.cc:1367] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:08:50.106547 26375 raft_consensus.cc:348] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b991e2c8414b6698e51851d34bc8f2" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 37205 } }
I20240501 14:08:50.106745 26375 raft_consensus.cc:374] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:50.106817 26375 raft_consensus.cc:729] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b991e2c8414b6698e51851d34bc8f2, State: Initialized, Role: FOLLOWER
I20240501 14:08:50.107234 26375 consensus_queue.cc:260] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b991e2c8414b6698e51851d34bc8f2" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 37205 } }
I20240501 14:08:50.107401 26375 raft_consensus.cc:388] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:50.107461 26375 raft_consensus.cc:482] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:50.107571 26375 raft_consensus.cc:3037] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:50.108700 26375 raft_consensus.cc:504] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b991e2c8414b6698e51851d34bc8f2" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 37205 } }
I20240501 14:08:50.109000 26375 leader_election.cc:302] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86b991e2c8414b6698e51851d34bc8f2; no voters:
I20240501 14:08:50.109300 26375 leader_election.cc:288] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:50.109392 26377 raft_consensus.cc:2781] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:50.109593 26377 raft_consensus.cc:686] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [term 1 LEADER]: Becoming Leader. State: Replica: 86b991e2c8414b6698e51851d34bc8f2, State: Running, Role: LEADER
I20240501 14:08:50.109861 26377 consensus_queue.cc:237] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b991e2c8414b6698e51851d34bc8f2" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 37205 } }
I20240501 14:08:50.109892 25968 heartbeater.cc:503] Master 127.22.252.190:44241 was elected leader, sending a full tablet report...
I20240501 14:08:50.110060 26375 ts_tablet_manager.cc:1398] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20240501 14:08:50.112749 25387 catalog_manager.cc:5484] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86b991e2c8414b6698e51851d34bc8f2 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "86b991e2c8414b6698e51851d34bc8f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b991e2c8414b6698e51851d34bc8f2" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 37205 } } }
I20240501 14:08:50.121042 23538 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 14:08:50.122161 23538 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e8f33be94f864561bec2e185b5a695c3 to finish bootstrapping
I20240501 14:08:50.127384 23538 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c6717727de344c0b90dd50fd5e70883d to finish bootstrapping
I20240501 14:08:50.132627 23538 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8a9c183caefa4437b53752c1203b8bb7 to finish bootstrapping
I20240501 14:08:50.137786 23538 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 86b991e2c8414b6698e51851d34bc8f2 to finish bootstrapping
I20240501 14:08:50.141930 23538 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e9e15d9915264a62a5c9e68f67872199 to finish bootstrapping
I20240501 14:08:50.146464 23538 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0e3ebe10573a4863b3734daf8f3c4382 to finish bootstrapping
I20240501 14:08:50.151546 23538 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3df2596a705349f185baa3b69dabfc25 to finish bootstrapping
I20240501 14:08:50.156585 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 25709
I20240501 14:08:50.162477 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 25840
I20240501 14:08:50.167737 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 25971
I20240501 14:08:50.172857 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 26102
I20240501 14:08:50.177779 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 26233
I20240501 14:08:50.220587 25706 heartbeater.cc:503] Master 127.22.252.190:44241 was elected leader, sending a full tablet report...
I20240501 14:08:53.934716 25387 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58550:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:08:53.947125 25640 tablet_service.cc:1467] Processing CreateTablet for tablet 9bc7d5eb523e4748b826531bc6be85ac (DEFAULT_TABLE table=rebalance_test_table_0 [id=aca60ec23c5145babbcbeaaa484c4348]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:53.947129 25508 tablet_service.cc:1467] Processing CreateTablet for tablet 36b4476375e546298f3d8184dda64204 (DEFAULT_TABLE table=rebalance_test_table_0 [id=aca60ec23c5145babbcbeaaa484c4348]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:53.947125 25509 tablet_service.cc:1467] Processing CreateTablet for tablet 8a1f7cc1afb143a8a132944682f969fe (DEFAULT_TABLE table=rebalance_test_table_0 [id=aca60ec23c5145babbcbeaaa484c4348]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:53.948191 25509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a1f7cc1afb143a8a132944682f969fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.948192 25640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bc7d5eb523e4748b826531bc6be85ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.948545 25508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36b4476375e546298f3d8184dda64204. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:53.953959 26397 tablet_bootstrap.cc:492] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3: Bootstrap starting.
I20240501 14:08:53.954032 26398 tablet_bootstrap.cc:492] T 9bc7d5eb523e4748b826531bc6be85ac P c6717727de344c0b90dd50fd5e70883d: Bootstrap starting.
I20240501 14:08:53.955874 26397 tablet_bootstrap.cc:654] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.955874 26398 tablet_bootstrap.cc:654] T 9bc7d5eb523e4748b826531bc6be85ac P c6717727de344c0b90dd50fd5e70883d: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.956568 26398 log.cc:824] T 9bc7d5eb523e4748b826531bc6be85ac P c6717727de344c0b90dd50fd5e70883d: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:53.956568 26397 log.cc:824] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:53.957980 26398 tablet_bootstrap.cc:492] T 9bc7d5eb523e4748b826531bc6be85ac P c6717727de344c0b90dd50fd5e70883d: No bootstrap required, opened a new log
I20240501 14:08:53.957980 26397 tablet_bootstrap.cc:492] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3: No bootstrap required, opened a new log
I20240501 14:08:53.958133 26397 ts_tablet_manager.cc:1367] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:08:53.958138 26398 ts_tablet_manager.cc:1367] T 9bc7d5eb523e4748b826531bc6be85ac P c6717727de344c0b90dd50fd5e70883d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:08:53.961095 26398 raft_consensus.cc:348] T 9bc7d5eb523e4748b826531bc6be85ac P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:53.961095 26397 raft_consensus.cc:348] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:53.961256 26398 raft_consensus.cc:374] T 9bc7d5eb523e4748b826531bc6be85ac P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.961266 26397 raft_consensus.cc:374] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.961295 26398 raft_consensus.cc:729] T 9bc7d5eb523e4748b826531bc6be85ac P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6717727de344c0b90dd50fd5e70883d, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.961334 26397 raft_consensus.cc:729] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f33be94f864561bec2e185b5a695c3, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.961717 26397 consensus_queue.cc:260] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:53.961716 26398 consensus_queue.cc:260] T 9bc7d5eb523e4748b826531bc6be85ac P c6717727de344c0b90dd50fd5e70883d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:53.961830 26398 raft_consensus.cc:388] T 9bc7d5eb523e4748b826531bc6be85ac P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:53.961833 26397 raft_consensus.cc:388] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:53.961896 26397 raft_consensus.cc:482] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:53.961899 26398 raft_consensus.cc:482] T 9bc7d5eb523e4748b826531bc6be85ac P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:53.962004 26398 raft_consensus.cc:3037] T 9bc7d5eb523e4748b826531bc6be85ac P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.962014 26397 raft_consensus.cc:3037] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.962674 26397 raft_consensus.cc:504] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:53.962677 26398 raft_consensus.cc:504] T 9bc7d5eb523e4748b826531bc6be85ac P c6717727de344c0b90dd50fd5e70883d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:53.962975 26398 leader_election.cc:302] T 9bc7d5eb523e4748b826531bc6be85ac P c6717727de344c0b90dd50fd5e70883d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6717727de344c0b90dd50fd5e70883d; no voters:
I20240501 14:08:53.962975 26397 leader_election.cc:302] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8f33be94f864561bec2e185b5a695c3; no voters:
I20240501 14:08:53.963232 26398 leader_election.cc:288] T 9bc7d5eb523e4748b826531bc6be85ac P c6717727de344c0b90dd50fd5e70883d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:53.963232 26397 leader_election.cc:288] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:53.963310 26402 raft_consensus.cc:2781] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:53.963308 26401 raft_consensus.cc:2781] T 9bc7d5eb523e4748b826531bc6be85ac P c6717727de344c0b90dd50fd5e70883d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:53.963491 26401 raft_consensus.cc:686] T 9bc7d5eb523e4748b826531bc6be85ac P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Becoming Leader. State: Replica: c6717727de344c0b90dd50fd5e70883d, State: Running, Role: LEADER
I20240501 14:08:53.963536 26402 raft_consensus.cc:686] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Becoming Leader. State: Replica: e8f33be94f864561bec2e185b5a695c3, State: Running, Role: LEADER
I20240501 14:08:53.963811 26401 consensus_queue.cc:237] T 9bc7d5eb523e4748b826531bc6be85ac P c6717727de344c0b90dd50fd5e70883d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:53.963816 26402 consensus_queue.cc:237] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:53.964043 26398 ts_tablet_manager.cc:1398] T 9bc7d5eb523e4748b826531bc6be85ac P c6717727de344c0b90dd50fd5e70883d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 14:08:53.964044 26397 ts_tablet_manager.cc:1398] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 14:08:53.964702 26397 tablet_bootstrap.cc:492] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3: Bootstrap starting.
I20240501 14:08:53.967236 26397 tablet_bootstrap.cc:654] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:53.967129 25386 catalog_manager.cc:5484] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8f33be94f864561bec2e185b5a695c3 (127.22.252.129). New cstate: current_term: 1 leader_uuid: "e8f33be94f864561bec2e185b5a695c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } }
I20240501 14:08:53.967191 25387 catalog_manager.cc:5484] T 9bc7d5eb523e4748b826531bc6be85ac P c6717727de344c0b90dd50fd5e70883d reported cstate change: term changed from 0 to 1, leader changed from <none> to c6717727de344c0b90dd50fd5e70883d (127.22.252.130). New cstate: current_term: 1 leader_uuid: "c6717727de344c0b90dd50fd5e70883d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } }
I20240501 14:08:53.968350 26397 tablet_bootstrap.cc:492] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3: No bootstrap required, opened a new log
I20240501 14:08:53.968436 26397 ts_tablet_manager.cc:1367] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:08:53.969106 26397 raft_consensus.cc:348] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:53.969679 26397 raft_consensus.cc:374] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:53.969740 26397 raft_consensus.cc:729] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f33be94f864561bec2e185b5a695c3, State: Initialized, Role: FOLLOWER
I20240501 14:08:53.969892 26397 consensus_queue.cc:260] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:53.970019 26397 raft_consensus.cc:388] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:53.970069 26397 raft_consensus.cc:482] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:53.970301 26397 raft_consensus.cc:3037] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:53.971284 26397 raft_consensus.cc:504] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:53.971412 26397 leader_election.cc:302] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8f33be94f864561bec2e185b5a695c3; no voters:
I20240501 14:08:53.971509 26397 leader_election.cc:288] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:53.971647 26406 raft_consensus.cc:2781] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:53.971724 26406 raft_consensus.cc:686] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Becoming Leader. State: Replica: e8f33be94f864561bec2e185b5a695c3, State: Running, Role: LEADER
I20240501 14:08:53.971840 26406 consensus_queue.cc:237] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:53.972383 26397 ts_tablet_manager.cc:1398] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:08:53.972990 25386 catalog_manager.cc:5484] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8f33be94f864561bec2e185b5a695c3 (127.22.252.129). New cstate: current_term: 1 leader_uuid: "e8f33be94f864561bec2e185b5a695c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } }
W20240501 14:08:54.086977 26414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:54.087198 26414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:54.090123 26414 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 60.5003 ms
time per row: 0.0302502 ms
I20240501 14:08:54.196141 25385 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58550:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:08:54.201378 25640 tablet_service.cc:1467] Processing CreateTablet for tablet 4e9029b14eda48cab26e246ed7535d65 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7f86e635baf845c191b673d7a95b523a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.201378 25639 tablet_service.cc:1467] Processing CreateTablet for tablet fd9a83b1cc1441da972fb41466aaf402 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7f86e635baf845c191b673d7a95b523a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.201503 25509 tablet_service.cc:1467] Processing CreateTablet for tablet 7d50344b10764e408b77e24aad669654 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7f86e635baf845c191b673d7a95b523a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.201751 25640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e9029b14eda48cab26e246ed7535d65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.201786 25509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d50344b10764e408b77e24aad669654. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.201892 25639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd9a83b1cc1441da972fb41466aaf402. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.203783 26398 tablet_bootstrap.cc:492] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d: Bootstrap starting.
I20240501 14:08:54.203999 26397 tablet_bootstrap.cc:492] T 7d50344b10764e408b77e24aad669654 P e8f33be94f864561bec2e185b5a695c3: Bootstrap starting.
I20240501 14:08:54.204658 26398 tablet_bootstrap.cc:654] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.204834 26397 tablet_bootstrap.cc:654] T 7d50344b10764e408b77e24aad669654 P e8f33be94f864561bec2e185b5a695c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.205757 26398 tablet_bootstrap.cc:492] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d: No bootstrap required, opened a new log
I20240501 14:08:54.205801 26397 tablet_bootstrap.cc:492] T 7d50344b10764e408b77e24aad669654 P e8f33be94f864561bec2e185b5a695c3: No bootstrap required, opened a new log
I20240501 14:08:54.205844 26398 ts_tablet_manager.cc:1367] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:08:54.205862 26397 ts_tablet_manager.cc:1367] T 7d50344b10764e408b77e24aad669654 P e8f33be94f864561bec2e185b5a695c3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:08:54.206207 26397 raft_consensus.cc:348] T 7d50344b10764e408b77e24aad669654 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.206290 26397 raft_consensus.cc:374] T 7d50344b10764e408b77e24aad669654 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.206311 26397 raft_consensus.cc:729] T 7d50344b10764e408b77e24aad669654 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f33be94f864561bec2e185b5a695c3, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.206287 26398 raft_consensus.cc:348] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.206394 26398 raft_consensus.cc:374] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.206429 26398 raft_consensus.cc:729] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6717727de344c0b90dd50fd5e70883d, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.206426 26397 consensus_queue.cc:260] T 7d50344b10764e408b77e24aad669654 P e8f33be94f864561bec2e185b5a695c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.206511 26397 raft_consensus.cc:388] T 7d50344b10764e408b77e24aad669654 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:54.206555 26397 raft_consensus.cc:482] T 7d50344b10764e408b77e24aad669654 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:54.206589 26397 raft_consensus.cc:3037] T 7d50344b10764e408b77e24aad669654 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.206561 26398 consensus_queue.cc:260] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.206645 26398 raft_consensus.cc:388] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:54.206688 26398 raft_consensus.cc:482] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:54.206738 26398 raft_consensus.cc:3037] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.207239 26397 raft_consensus.cc:504] T 7d50344b10764e408b77e24aad669654 P e8f33be94f864561bec2e185b5a695c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.207361 26397 leader_election.cc:302] T 7d50344b10764e408b77e24aad669654 P e8f33be94f864561bec2e185b5a695c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8f33be94f864561bec2e185b5a695c3; no voters:
I20240501 14:08:54.207357 26398 raft_consensus.cc:504] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.207479 26398 leader_election.cc:302] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6717727de344c0b90dd50fd5e70883d; no voters:
I20240501 14:08:54.207561 26398 leader_election.cc:288] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:54.207613 26401 raft_consensus.cc:2781] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:54.207698 26401 raft_consensus.cc:686] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Becoming Leader. State: Replica: c6717727de344c0b90dd50fd5e70883d, State: Running, Role: LEADER
I20240501 14:08:54.207803 26401 consensus_queue.cc:237] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.208122 26398 ts_tablet_manager.cc:1398] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:08:54.208609 26398 tablet_bootstrap.cc:492] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d: Bootstrap starting.
I20240501 14:08:54.209089 25385 catalog_manager.cc:5484] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d reported cstate change: term changed from 0 to 1, leader changed from <none> to c6717727de344c0b90dd50fd5e70883d (127.22.252.130). New cstate: current_term: 1 leader_uuid: "c6717727de344c0b90dd50fd5e70883d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } }
I20240501 14:08:54.209266 26398 tablet_bootstrap.cc:654] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.210304 26398 tablet_bootstrap.cc:492] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d: No bootstrap required, opened a new log
I20240501 14:08:54.210376 26398 ts_tablet_manager.cc:1367] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:54.210739 26398 raft_consensus.cc:348] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.210799 26398 raft_consensus.cc:374] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.210822 26398 raft_consensus.cc:729] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6717727de344c0b90dd50fd5e70883d, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.210913 26398 consensus_queue.cc:260] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.210974 26398 raft_consensus.cc:388] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:54.211009 26398 raft_consensus.cc:482] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:54.211050 26398 raft_consensus.cc:3037] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.211638 26398 raft_consensus.cc:504] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.211752 26398 leader_election.cc:302] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6717727de344c0b90dd50fd5e70883d; no voters:
I20240501 14:08:54.211820 26398 leader_election.cc:288] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:54.211889 26404 raft_consensus.cc:2781] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:54.211953 26404 raft_consensus.cc:686] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Becoming Leader. State: Replica: c6717727de344c0b90dd50fd5e70883d, State: Running, Role: LEADER
I20240501 14:08:54.212035 26398 ts_tablet_manager.cc:1398] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:08:54.212061 26404 consensus_queue.cc:237] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.212657 26397 leader_election.cc:288] T 7d50344b10764e408b77e24aad669654 P e8f33be94f864561bec2e185b5a695c3 [CANDIDATE]: Term 1 election: Requested vote from peers
W20240501 14:08:54.212761 25707 tablet.cc:2369] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:08:54.212885 26397 ts_tablet_manager.cc:1398] T 7d50344b10764e408b77e24aad669654 P e8f33be94f864561bec2e185b5a695c3: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240501 14:08:54.213114 26403 raft_consensus.cc:2781] T 7d50344b10764e408b77e24aad669654 P e8f33be94f864561bec2e185b5a695c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:54.213228 26403 raft_consensus.cc:686] T 7d50344b10764e408b77e24aad669654 P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Becoming Leader. State: Replica: e8f33be94f864561bec2e185b5a695c3, State: Running, Role: LEADER
I20240501 14:08:54.213363 26403 consensus_queue.cc:237] T 7d50344b10764e408b77e24aad669654 P e8f33be94f864561bec2e185b5a695c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.215070 25387 catalog_manager.cc:5484] T 7d50344b10764e408b77e24aad669654 P e8f33be94f864561bec2e185b5a695c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8f33be94f864561bec2e185b5a695c3 (127.22.252.129). New cstate: current_term: 1 leader_uuid: "e8f33be94f864561bec2e185b5a695c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } }
I20240501 14:08:54.216786 25385 catalog_manager.cc:5484] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d reported cstate change: term changed from 0 to 1, leader changed from <none> to c6717727de344c0b90dd50fd5e70883d (127.22.252.130). New cstate: current_term: 1 leader_uuid: "c6717727de344c0b90dd50fd5e70883d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } }
W20240501 14:08:54.323874 26458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:54.324117 26458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:54.327157 26458 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 60.0836 ms
time per row: 0.0300418 ms
I20240501 14:08:54.426369 25380 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58550:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:08:54.431821 25509 tablet_service.cc:1467] Processing CreateTablet for tablet c230502585ed4dd1aa75bc02a3e7d242 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ec7f7fdeebf748848f7e012922717645]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.431921 25640 tablet_service.cc:1467] Processing CreateTablet for tablet 50a6b368aa5e414c9a6464b99f5ea51f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ec7f7fdeebf748848f7e012922717645]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.431999 25508 tablet_service.cc:1467] Processing CreateTablet for tablet 90021b467e724c03850f3718c014742e (DEFAULT_TABLE table=rebalance_test_table_2 [id=ec7f7fdeebf748848f7e012922717645]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.432126 25509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c230502585ed4dd1aa75bc02a3e7d242. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.432262 25640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50a6b368aa5e414c9a6464b99f5ea51f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.432314 25508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90021b467e724c03850f3718c014742e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.433826 26397 tablet_bootstrap.cc:492] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3: Bootstrap starting.
I20240501 14:08:54.434223 26398 tablet_bootstrap.cc:492] T 50a6b368aa5e414c9a6464b99f5ea51f P c6717727de344c0b90dd50fd5e70883d: Bootstrap starting.
I20240501 14:08:54.434736 26397 tablet_bootstrap.cc:654] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.435104 26398 tablet_bootstrap.cc:654] T 50a6b368aa5e414c9a6464b99f5ea51f P c6717727de344c0b90dd50fd5e70883d: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.435945 26397 tablet_bootstrap.cc:492] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3: No bootstrap required, opened a new log
I20240501 14:08:54.436043 26397 ts_tablet_manager.cc:1367] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:54.436226 26398 tablet_bootstrap.cc:492] T 50a6b368aa5e414c9a6464b99f5ea51f P c6717727de344c0b90dd50fd5e70883d: No bootstrap required, opened a new log
I20240501 14:08:54.436342 26398 ts_tablet_manager.cc:1367] T 50a6b368aa5e414c9a6464b99f5ea51f P c6717727de344c0b90dd50fd5e70883d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:54.436561 26397 raft_consensus.cc:348] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.436668 26397 raft_consensus.cc:374] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.436709 26397 raft_consensus.cc:729] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f33be94f864561bec2e185b5a695c3, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.436810 26398 raft_consensus.cc:348] T 50a6b368aa5e414c9a6464b99f5ea51f P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.436836 26397 consensus_queue.cc:260] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.436902 26398 raft_consensus.cc:374] T 50a6b368aa5e414c9a6464b99f5ea51f P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.436914 26397 raft_consensus.cc:388] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:54.436946 26398 raft_consensus.cc:729] T 50a6b368aa5e414c9a6464b99f5ea51f P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6717727de344c0b90dd50fd5e70883d, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.436952 26397 raft_consensus.cc:482] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:54.436998 26397 raft_consensus.cc:3037] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.437067 26398 consensus_queue.cc:260] T 50a6b368aa5e414c9a6464b99f5ea51f P c6717727de344c0b90dd50fd5e70883d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.437161 26398 raft_consensus.cc:388] T 50a6b368aa5e414c9a6464b99f5ea51f P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:54.437194 26398 raft_consensus.cc:482] T 50a6b368aa5e414c9a6464b99f5ea51f P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:54.437232 26398 raft_consensus.cc:3037] T 50a6b368aa5e414c9a6464b99f5ea51f P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.437871 26397 raft_consensus.cc:504] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.438024 26397 leader_election.cc:302] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8f33be94f864561bec2e185b5a695c3; no voters:
I20240501 14:08:54.438148 26397 leader_election.cc:288] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:54.438118 26398 raft_consensus.cc:504] T 50a6b368aa5e414c9a6464b99f5ea51f P c6717727de344c0b90dd50fd5e70883d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.438205 26406 raft_consensus.cc:2781] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:54.438258 26398 leader_election.cc:302] T 50a6b368aa5e414c9a6464b99f5ea51f P c6717727de344c0b90dd50fd5e70883d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6717727de344c0b90dd50fd5e70883d; no voters:
I20240501 14:08:54.438323 26406 raft_consensus.cc:686] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Becoming Leader. State: Replica: e8f33be94f864561bec2e185b5a695c3, State: Running, Role: LEADER
I20240501 14:08:54.438401 26401 raft_consensus.cc:2781] T 50a6b368aa5e414c9a6464b99f5ea51f P c6717727de344c0b90dd50fd5e70883d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:54.438464 26401 raft_consensus.cc:686] T 50a6b368aa5e414c9a6464b99f5ea51f P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Becoming Leader. State: Replica: c6717727de344c0b90dd50fd5e70883d, State: Running, Role: LEADER
I20240501 14:08:54.438434 26406 consensus_queue.cc:237] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.438548 26397 ts_tablet_manager.cc:1398] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:08:54.438634 26401 consensus_queue.cc:237] T 50a6b368aa5e414c9a6464b99f5ea51f P c6717727de344c0b90dd50fd5e70883d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.438694 26397 tablet_bootstrap.cc:492] T 90021b467e724c03850f3718c014742e P e8f33be94f864561bec2e185b5a695c3: Bootstrap starting.
I20240501 14:08:54.439539 26397 tablet_bootstrap.cc:654] T 90021b467e724c03850f3718c014742e P e8f33be94f864561bec2e185b5a695c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.439750 26398 leader_election.cc:288] T 50a6b368aa5e414c9a6464b99f5ea51f P c6717727de344c0b90dd50fd5e70883d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:54.439731 25380 catalog_manager.cc:5484] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8f33be94f864561bec2e185b5a695c3 (127.22.252.129). New cstate: current_term: 1 leader_uuid: "e8f33be94f864561bec2e185b5a695c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } }
I20240501 14:08:54.439949 25387 catalog_manager.cc:5484] T 50a6b368aa5e414c9a6464b99f5ea51f P c6717727de344c0b90dd50fd5e70883d reported cstate change: term changed from 0 to 1, leader changed from <none> to c6717727de344c0b90dd50fd5e70883d (127.22.252.130). New cstate: current_term: 1 leader_uuid: "c6717727de344c0b90dd50fd5e70883d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } }
I20240501 14:08:54.440037 26398 ts_tablet_manager.cc:1398] T 50a6b368aa5e414c9a6464b99f5ea51f P c6717727de344c0b90dd50fd5e70883d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:08:54.441066 26397 tablet_bootstrap.cc:492] T 90021b467e724c03850f3718c014742e P e8f33be94f864561bec2e185b5a695c3: No bootstrap required, opened a new log
I20240501 14:08:54.441148 26397 ts_tablet_manager.cc:1367] T 90021b467e724c03850f3718c014742e P e8f33be94f864561bec2e185b5a695c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:54.441587 26397 raft_consensus.cc:348] T 90021b467e724c03850f3718c014742e P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.441686 26397 raft_consensus.cc:374] T 90021b467e724c03850f3718c014742e P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.441725 26397 raft_consensus.cc:729] T 90021b467e724c03850f3718c014742e P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f33be94f864561bec2e185b5a695c3, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.441838 26397 consensus_queue.cc:260] T 90021b467e724c03850f3718c014742e P e8f33be94f864561bec2e185b5a695c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.441916 26397 raft_consensus.cc:388] T 90021b467e724c03850f3718c014742e P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:54.441948 26397 raft_consensus.cc:482] T 90021b467e724c03850f3718c014742e P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:54.441989 26397 raft_consensus.cc:3037] T 90021b467e724c03850f3718c014742e P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.442817 26397 raft_consensus.cc:504] T 90021b467e724c03850f3718c014742e P e8f33be94f864561bec2e185b5a695c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.442919 26397 leader_election.cc:302] T 90021b467e724c03850f3718c014742e P e8f33be94f864561bec2e185b5a695c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8f33be94f864561bec2e185b5a695c3; no voters:
I20240501 14:08:54.443022 26397 leader_election.cc:288] T 90021b467e724c03850f3718c014742e P e8f33be94f864561bec2e185b5a695c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:54.443240 26397 ts_tablet_manager.cc:1398] T 90021b467e724c03850f3718c014742e P e8f33be94f864561bec2e185b5a695c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:54.443392 26406 raft_consensus.cc:2781] T 90021b467e724c03850f3718c014742e P e8f33be94f864561bec2e185b5a695c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:54.443475 26406 raft_consensus.cc:686] T 90021b467e724c03850f3718c014742e P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Becoming Leader. State: Replica: e8f33be94f864561bec2e185b5a695c3, State: Running, Role: LEADER
I20240501 14:08:54.443584 26406 consensus_queue.cc:237] T 90021b467e724c03850f3718c014742e P e8f33be94f864561bec2e185b5a695c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.445065 25387 catalog_manager.cc:5484] T 90021b467e724c03850f3718c014742e P e8f33be94f864561bec2e185b5a695c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8f33be94f864561bec2e185b5a695c3 (127.22.252.129). New cstate: current_term: 1 leader_uuid: "e8f33be94f864561bec2e185b5a695c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } }
W20240501 14:08:54.555020 26491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:54.555239 26491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:54.558218 26491 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 69.0872 ms
time per row: 0.0345436 ms
I20240501 14:08:54.666533 25387 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58550:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:08:54.671730 25640 tablet_service.cc:1467] Processing CreateTablet for tablet 82731ca00eea49248cf51765e441fea6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=af481f6478684cf7ab01dc10b39463a7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.671777 25639 tablet_service.cc:1467] Processing CreateTablet for tablet a58883c2a8074e01b2170310e3008e52 (DEFAULT_TABLE table=rebalance_test_table_3 [id=af481f6478684cf7ab01dc10b39463a7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.672036 25640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82731ca00eea49248cf51765e441fea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.671974 25508 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae562f0735b4afeb52172957637cbcf (DEFAULT_TABLE table=rebalance_test_table_3 [id=af481f6478684cf7ab01dc10b39463a7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.672148 25639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a58883c2a8074e01b2170310e3008e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.672281 25508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae562f0735b4afeb52172957637cbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.673741 26398 tablet_bootstrap.cc:492] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d: Bootstrap starting.
I20240501 14:08:54.674290 26397 tablet_bootstrap.cc:492] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3: Bootstrap starting.
I20240501 14:08:54.674683 26398 tablet_bootstrap.cc:654] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.675083 26397 tablet_bootstrap.cc:654] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.675627 26398 tablet_bootstrap.cc:492] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d: No bootstrap required, opened a new log
I20240501 14:08:54.675724 26398 ts_tablet_manager.cc:1367] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:54.675990 26397 tablet_bootstrap.cc:492] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3: No bootstrap required, opened a new log
I20240501 14:08:54.676056 26397 ts_tablet_manager.cc:1367] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:54.676107 26398 raft_consensus.cc:348] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.676204 26398 raft_consensus.cc:374] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.676267 26398 raft_consensus.cc:729] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6717727de344c0b90dd50fd5e70883d, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.676393 26398 consensus_queue.cc:260] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.676489 26398 raft_consensus.cc:388] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:54.676534 26398 raft_consensus.cc:482] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:54.676497 26397 raft_consensus.cc:348] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.676585 26398 raft_consensus.cc:3037] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.676586 26397 raft_consensus.cc:374] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.676620 26397 raft_consensus.cc:729] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f33be94f864561bec2e185b5a695c3, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.676741 26397 consensus_queue.cc:260] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.676805 26397 raft_consensus.cc:388] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:54.676826 26397 raft_consensus.cc:482] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:54.676874 26397 raft_consensus.cc:3037] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.677310 26398 raft_consensus.cc:504] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.677436 26398 leader_election.cc:302] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6717727de344c0b90dd50fd5e70883d; no voters:
I20240501 14:08:54.677520 26398 leader_election.cc:288] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:54.677511 26397 raft_consensus.cc:504] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.677629 26397 leader_election.cc:302] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8f33be94f864561bec2e185b5a695c3; no voters:
I20240501 14:08:54.677712 26397 leader_election.cc:288] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:54.677737 26398 ts_tablet_manager.cc:1398] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:54.677848 26403 raft_consensus.cc:2781] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:54.677912 26398 tablet_bootstrap.cc:492] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d: Bootstrap starting.
I20240501 14:08:54.677932 26403 raft_consensus.cc:686] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Becoming Leader. State: Replica: e8f33be94f864561bec2e185b5a695c3, State: Running, Role: LEADER
I20240501 14:08:54.678066 26403 consensus_queue.cc:237] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.678683 26398 tablet_bootstrap.cc:654] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.677927 26397 ts_tablet_manager.cc:1398] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:54.679270 26405 raft_consensus.cc:2781] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:54.679354 26405 raft_consensus.cc:686] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Becoming Leader. State: Replica: c6717727de344c0b90dd50fd5e70883d, State: Running, Role: LEADER
I20240501 14:08:54.679469 26405 consensus_queue.cc:237] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.679580 25385 catalog_manager.cc:5484] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8f33be94f864561bec2e185b5a695c3 (127.22.252.129). New cstate: current_term: 1 leader_uuid: "e8f33be94f864561bec2e185b5a695c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } }
I20240501 14:08:54.679874 26398 tablet_bootstrap.cc:492] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d: No bootstrap required, opened a new log
I20240501 14:08:54.679958 26398 ts_tablet_manager.cc:1367] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:54.680970 26398 raft_consensus.cc:348] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.681066 26398 raft_consensus.cc:374] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.681109 26398 raft_consensus.cc:729] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6717727de344c0b90dd50fd5e70883d, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.681224 26398 consensus_queue.cc:260] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.681324 26398 raft_consensus.cc:388] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:54.681356 26398 raft_consensus.cc:482] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:54.681401 26398 raft_consensus.cc:3037] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.682164 26398 raft_consensus.cc:504] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.682314 26398 leader_election.cc:302] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6717727de344c0b90dd50fd5e70883d; no voters:
I20240501 14:08:54.682410 26398 leader_election.cc:288] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:54.682467 26454 raft_consensus.cc:2781] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:54.682562 26454 raft_consensus.cc:686] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Becoming Leader. State: Replica: c6717727de344c0b90dd50fd5e70883d, State: Running, Role: LEADER
I20240501 14:08:54.682626 26398 ts_tablet_manager.cc:1398] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:08:54.682461 25387 catalog_manager.cc:5484] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d reported cstate change: term changed from 0 to 1, leader changed from <none> to c6717727de344c0b90dd50fd5e70883d (127.22.252.130). New cstate: current_term: 1 leader_uuid: "c6717727de344c0b90dd50fd5e70883d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } }
I20240501 14:08:54.682672 26454 consensus_queue.cc:237] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.685326 25387 catalog_manager.cc:5484] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d reported cstate change: term changed from 0 to 1, leader changed from <none> to c6717727de344c0b90dd50fd5e70883d (127.22.252.130). New cstate: current_term: 1 leader_uuid: "c6717727de344c0b90dd50fd5e70883d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } }
W20240501 14:08:54.790300 26522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:54.790531 26522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:54.793404 26522 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 76.074 ms
time per row: 0.038037 ms
I20240501 14:08:54.909806 25378 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58550:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:08:54.914923 25508 tablet_service.cc:1467] Processing CreateTablet for tablet 78ff5bcb4c5445659d365e84dc810ebc (DEFAULT_TABLE table=rebalance_test_table_4 [id=0622125ee1f04076a30c4aa37417bf64]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.914994 25509 tablet_service.cc:1467] Processing CreateTablet for tablet d16f3d9f72b44c2cb15a6c634bee2a28 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0622125ee1f04076a30c4aa37417bf64]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.915103 25640 tablet_service.cc:1467] Processing CreateTablet for tablet 4dfd32332f154be489d3a15a2192d07a (DEFAULT_TABLE table=rebalance_test_table_4 [id=0622125ee1f04076a30c4aa37417bf64]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:54.915307 25509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d16f3d9f72b44c2cb15a6c634bee2a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.915412 25640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dfd32332f154be489d3a15a2192d07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.915499 25508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ff5bcb4c5445659d365e84dc810ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.917382 26398 tablet_bootstrap.cc:492] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d: Bootstrap starting.
I20240501 14:08:54.917675 26397 tablet_bootstrap.cc:492] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3: Bootstrap starting.
I20240501 14:08:54.918206 26398 tablet_bootstrap.cc:654] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.918440 26397 tablet_bootstrap.cc:654] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.919272 26398 tablet_bootstrap.cc:492] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d: No bootstrap required, opened a new log
I20240501 14:08:54.919361 26398 ts_tablet_manager.cc:1367] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:54.919536 26397 tablet_bootstrap.cc:492] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3: No bootstrap required, opened a new log
I20240501 14:08:54.919616 26397 ts_tablet_manager.cc:1367] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:54.919762 26398 raft_consensus.cc:348] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.919845 26398 raft_consensus.cc:374] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.919867 26398 raft_consensus.cc:729] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6717727de344c0b90dd50fd5e70883d, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.919988 26398 consensus_queue.cc:260] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.920051 26398 raft_consensus.cc:388] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:54.920074 26398 raft_consensus.cc:482] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:54.920045 26397 raft_consensus.cc:348] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.920121 26398 raft_consensus.cc:3037] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.920152 26397 raft_consensus.cc:374] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.920193 26397 raft_consensus.cc:729] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f33be94f864561bec2e185b5a695c3, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.920343 26397 consensus_queue.cc:260] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.920421 26397 raft_consensus.cc:388] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:54.920459 26397 raft_consensus.cc:482] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:54.920507 26397 raft_consensus.cc:3037] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.920876 26398 raft_consensus.cc:504] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.920997 26398 leader_election.cc:302] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6717727de344c0b90dd50fd5e70883d; no voters:
I20240501 14:08:54.921092 26398 leader_election.cc:288] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:54.921154 26454 raft_consensus.cc:2781] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:54.921217 26397 raft_consensus.cc:504] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.921281 26454 raft_consensus.cc:686] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Becoming Leader. State: Replica: c6717727de344c0b90dd50fd5e70883d, State: Running, Role: LEADER
I20240501 14:08:54.921337 26398 ts_tablet_manager.cc:1398] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:08:54.921360 26397 leader_election.cc:302] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8f33be94f864561bec2e185b5a695c3; no voters:
I20240501 14:08:54.921398 26454 consensus_queue.cc:237] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:54.921491 26402 raft_consensus.cc:2781] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:54.921561 26402 raft_consensus.cc:686] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Becoming Leader. State: Replica: e8f33be94f864561bec2e185b5a695c3, State: Running, Role: LEADER
I20240501 14:08:54.921658 26402 consensus_queue.cc:237] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.922488 26397 leader_election.cc:288] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:54.922786 26397 ts_tablet_manager.cc:1398] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:08:54.922758 25378 catalog_manager.cc:5484] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d reported cstate change: term changed from 0 to 1, leader changed from <none> to c6717727de344c0b90dd50fd5e70883d (127.22.252.130). New cstate: current_term: 1 leader_uuid: "c6717727de344c0b90dd50fd5e70883d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } }
I20240501 14:08:54.922925 26397 tablet_bootstrap.cc:492] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3: Bootstrap starting.
I20240501 14:08:54.922935 25387 catalog_manager.cc:5484] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8f33be94f864561bec2e185b5a695c3 (127.22.252.129). New cstate: current_term: 1 leader_uuid: "e8f33be94f864561bec2e185b5a695c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } }
I20240501 14:08:54.923696 26397 tablet_bootstrap.cc:654] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:54.925045 26397 tablet_bootstrap.cc:492] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3: No bootstrap required, opened a new log
I20240501 14:08:54.925108 26397 ts_tablet_manager.cc:1367] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:08:54.925501 26397 raft_consensus.cc:348] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.925590 26397 raft_consensus.cc:374] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:54.925622 26397 raft_consensus.cc:729] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f33be94f864561bec2e185b5a695c3, State: Initialized, Role: FOLLOWER
I20240501 14:08:54.925729 26397 consensus_queue.cc:260] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.925809 26397 raft_consensus.cc:388] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:54.925848 26397 raft_consensus.cc:482] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:54.925899 26397 raft_consensus.cc:3037] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:54.926523 26397 raft_consensus.cc:504] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.926631 26397 leader_election.cc:302] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8f33be94f864561bec2e185b5a695c3; no voters:
I20240501 14:08:54.926710 26397 leader_election.cc:288] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:54.926898 26397 ts_tablet_manager.cc:1398] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:54.927047 26402 raft_consensus.cc:2781] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:54.927136 26402 raft_consensus.cc:686] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Becoming Leader. State: Replica: e8f33be94f864561bec2e185b5a695c3, State: Running, Role: LEADER
I20240501 14:08:54.927242 26402 consensus_queue.cc:237] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:54.929026 25378 catalog_manager.cc:5484] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8f33be94f864561bec2e185b5a695c3 (127.22.252.129). New cstate: current_term: 1 leader_uuid: "e8f33be94f864561bec2e185b5a695c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } }
W20240501 14:08:55.036556 26553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:55.036784 26553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:55.039775 26553 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 56.3753 ms
time per row: 0.0281877 ms
I20240501 14:08:55.138490 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.131:36947
--local_ip_for_outbound_sockets=127.22.252.131
--tserver_master_addrs=127.22.252.190:44241
--webserver_port=40119
--webserver_interface=127.22.252.131
--builtin_ntp_servers=127.22.252.148:35091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:08:55.251056 26578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:55.251281 26578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:55.251322 26578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:55.254362 26578 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:08:55.254418 26578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:55.254480 26578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.131
I20240501 14:08:55.257894 26578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:35091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.131:36947
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.252.131
--webserver_port=40119
--tserver_master_addrs=127.22.252.190:44241
--never_fsync=true
--heap_profile_path=/tmp/kudu.26578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.131
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:55.258692 26578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:55.259750 26578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:55.267969 26583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:55.268738 26584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:55.269461 26586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:55.270464 26578 server_base.cc:1034] running on GCE node
I20240501 14:08:55.270814 26578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:55.271353 26578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:55.272512 26578 hybrid_clock.cc:648] HybridClock initialized: now 1714572535272496 us; error 30 us; skew 500 ppm
I20240501 14:08:55.274339 26578 webserver.cc:460] Webserver started at http://127.22.252.131:40119/ using document root <none> and password file <none>
I20240501 14:08:55.274932 26578 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:55.275019 26578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:55.278347 26578 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20240501 14:08:55.280149 26592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:55.281126 26578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 14:08:55.281232 26578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8a9c183caefa4437b53752c1203b8bb7"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-bkzb"
I20240501 14:08:55.281616 26578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:55.296548 26578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:55.297080 26578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:55.297217 26578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:55.297708 26578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:55.298615 26578 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:55.298666 26578 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:55.298720 26578 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:55.298751 26578 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:55.307013 26578 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.131:36947
I20240501 14:08:55.307036 26705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.131:36947 every 8 connection(s)
I20240501 14:08:55.308043 26578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:08:55.313920 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 26578
I20240501 14:08:55.314316 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.132:37205
--local_ip_for_outbound_sockets=127.22.252.132
--tserver_master_addrs=127.22.252.190:44241
--webserver_port=34309
--webserver_interface=127.22.252.132
--builtin_ntp_servers=127.22.252.148:35091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:08:55.316607 26706 heartbeater.cc:347] Connected to a master server at 127.22.252.190:44241
I20240501 14:08:55.316833 26706 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:55.317291 26706 heartbeater.cc:511] Master 127.22.252.190:44241 requested a full tablet report, sending...
I20240501 14:08:55.318363 25387 ts_manager.cc:194] Re-registered known tserver with Master: 8a9c183caefa4437b53752c1203b8bb7 (127.22.252.131:36947)
I20240501 14:08:55.318898 25387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.131:51073
W20240501 14:08:55.418329 26709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:55.418520 26709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:55.418591 26709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:55.421619 26709 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:08:55.421684 26709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:55.421774 26709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.132
I20240501 14:08:55.425272 26709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:35091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.132:37205
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.252.132
--webserver_port=34309
--tserver_master_addrs=127.22.252.190:44241
--never_fsync=true
--heap_profile_path=/tmp/kudu.26709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.132
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:55.426136 26709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:55.427228 26709 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:55.435184 26714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:55.436131 26717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:55.436857 26715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:55.437646 26709 server_base.cc:1034] running on GCE node
I20240501 14:08:55.437988 26709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:55.438511 26709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:55.439656 26709 hybrid_clock.cc:648] HybridClock initialized: now 1714572535439636 us; error 30 us; skew 500 ppm
I20240501 14:08:55.441330 26709 webserver.cc:460] Webserver started at http://127.22.252.132:34309/ using document root <none> and password file <none>
I20240501 14:08:55.441839 26709 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:55.441913 26709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:55.445094 26709 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 14:08:55.446878 26723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:55.447783 26709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:55.447921 26709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "86b991e2c8414b6698e51851d34bc8f2"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-bkzb"
I20240501 14:08:55.448345 26709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:55.457557 26709 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:55.458108 26709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:55.458266 26709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:55.458724 26709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:55.459729 26730 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240501 14:08:55.462464 26709 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 14:08:55.462534 26709 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240501 14:08:55.462592 26709 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 14:08:55.463900 26709 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 14:08:55.463943 26709 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240501 14:08:55.464143 26730 tablet_bootstrap.cc:492] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2: Bootstrap starting.
I20240501 14:08:55.473511 26709 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.132:37205
I20240501 14:08:55.473553 26837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.132:37205 every 8 connection(s)
I20240501 14:08:55.474467 26709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:08:55.477437 26730 log.cc:824] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:55.479540 26730 tablet_bootstrap.cc:492] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:08:55.479843 26730 tablet_bootstrap.cc:492] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2: Bootstrap complete.
I20240501 14:08:55.480087 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 26709
I20240501 14:08:55.480513 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.133:42377
--local_ip_for_outbound_sockets=127.22.252.133
--tserver_master_addrs=127.22.252.190:44241
--webserver_port=35765
--webserver_interface=127.22.252.133
--builtin_ntp_servers=127.22.252.148:35091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:08:55.480957 26730 ts_tablet_manager.cc:1367] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.006s
I20240501 14:08:55.482955 26838 heartbeater.cc:347] Connected to a master server at 127.22.252.190:44241
I20240501 14:08:55.483161 26838 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:55.483630 26838 heartbeater.cc:511] Master 127.22.252.190:44241 requested a full tablet report, sending...
I20240501 14:08:55.483661 26730 raft_consensus.cc:348] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b991e2c8414b6698e51851d34bc8f2" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 37205 } }
I20240501 14:08:55.483975 26730 raft_consensus.cc:729] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b991e2c8414b6698e51851d34bc8f2, State: Initialized, Role: FOLLOWER
I20240501 14:08:55.484452 26730 consensus_queue.cc:260] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b991e2c8414b6698e51851d34bc8f2" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 37205 } }
I20240501 14:08:55.484586 26730 raft_consensus.cc:388] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:55.484645 26730 raft_consensus.cc:482] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:55.484748 26730 raft_consensus.cc:3037] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:08:55.486325 26730 raft_consensus.cc:504] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b991e2c8414b6698e51851d34bc8f2" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 37205 } }
I20240501 14:08:55.486654 26730 leader_election.cc:302] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86b991e2c8414b6698e51851d34bc8f2; no voters:
I20240501 14:08:55.486908 26730 leader_election.cc:288] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240501 14:08:55.487030 26843 raft_consensus.cc:2781] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:08:55.487217 26843 raft_consensus.cc:686] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [term 2 LEADER]: Becoming Leader. State: Replica: 86b991e2c8414b6698e51851d34bc8f2, State: Running, Role: LEADER
I20240501 14:08:55.487522 26843 consensus_queue.cc:237] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b991e2c8414b6698e51851d34bc8f2" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 37205 } }
I20240501 14:08:55.487725 26730 ts_tablet_manager.cc:1398] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20240501 14:08:55.488790 25387 ts_manager.cc:194] Re-registered known tserver with Master: 86b991e2c8414b6698e51851d34bc8f2 (127.22.252.132:37205)
I20240501 14:08:55.489845 25387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.132:52149
I20240501 14:08:55.491894 26838 heartbeater.cc:503] Master 127.22.252.190:44241 was elected leader, sending a full tablet report...
I20240501 14:08:55.492888 25387 catalog_manager.cc:5484] T 844ba8b3ff6c4d88a101d1c38e7ad0b3 P 86b991e2c8414b6698e51851d34bc8f2 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "86b991e2c8414b6698e51851d34bc8f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b991e2c8414b6698e51851d34bc8f2" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 37205 } } }
W20240501 14:08:55.583990 26842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:55.584192 26842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:55.584264 26842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:55.587424 26842 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:08:55.587476 26842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:55.587530 26842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.133
I20240501 14:08:55.592010 26842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:35091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.133:42377
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.252.133
--webserver_port=35765
--tserver_master_addrs=127.22.252.190:44241
--never_fsync=true
--heap_profile_path=/tmp/kudu.26842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.133
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:55.592985 26842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:55.594240 26842 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:55.604091 26858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:55.604597 26855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:55.605312 26856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:55.606489 26842 server_base.cc:1034] running on GCE node
I20240501 14:08:55.606823 26842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:55.607332 26842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:55.608479 26842 hybrid_clock.cc:648] HybridClock initialized: now 1714572535608455 us; error 34 us; skew 500 ppm
I20240501 14:08:55.610144 26842 webserver.cc:460] Webserver started at http://127.22.252.133:35765/ using document root <none> and password file <none>
I20240501 14:08:55.610657 26842 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:55.610728 26842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:55.613909 26842 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:08:55.615502 26864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:55.616475 26842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:55.616590 26842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e9e15d9915264a62a5c9e68f67872199"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-bkzb"
I20240501 14:08:55.617017 26842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:55.653017 26842 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:55.653699 26842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:55.653858 26842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:55.654388 26842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:55.655264 26842 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:55.655313 26842 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:55.655371 26842 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:55.655396 26842 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:55.663484 26842 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.133:42377
I20240501 14:08:55.663507 26977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.133:42377 every 8 connection(s)
I20240501 14:08:55.664481 26842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:08:55.669605 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 26842
I20240501 14:08:55.670024 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.134:42583
--local_ip_for_outbound_sockets=127.22.252.134
--tserver_master_addrs=127.22.252.190:44241
--webserver_port=43823
--webserver_interface=127.22.252.134
--builtin_ntp_servers=127.22.252.148:35091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:08:55.673091 26978 heartbeater.cc:347] Connected to a master server at 127.22.252.190:44241
I20240501 14:08:55.673293 26978 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:55.673724 26978 heartbeater.cc:511] Master 127.22.252.190:44241 requested a full tablet report, sending...
I20240501 14:08:55.674697 25387 ts_manager.cc:194] Re-registered known tserver with Master: e9e15d9915264a62a5c9e68f67872199 (127.22.252.133:42377)
I20240501 14:08:55.675487 25387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.133:39863
W20240501 14:08:55.772167 26981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:55.772387 26981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:55.772457 26981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:55.775336 26981 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:08:55.775399 26981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:55.775486 26981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.134
I20240501 14:08:55.778942 26981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:35091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.134:42583
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.252.134
--webserver_port=43823
--tserver_master_addrs=127.22.252.190:44241
--never_fsync=true
--heap_profile_path=/tmp/kudu.26981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.134
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:55.779755 26981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:55.780962 26981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:55.789068 26989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:55.789832 26986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:55.790865 26987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:55.791788 26981 server_base.cc:1034] running on GCE node
I20240501 14:08:55.792116 26981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:55.792635 26981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:55.793797 26981 hybrid_clock.cc:648] HybridClock initialized: now 1714572535793779 us; error 37 us; skew 500 ppm
I20240501 14:08:55.795500 26981 webserver.cc:460] Webserver started at http://127.22.252.134:43823/ using document root <none> and password file <none>
I20240501 14:08:55.796044 26981 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:55.796136 26981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:55.799331 26981 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:55.801155 26995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:55.802067 26981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:55.802166 26981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "0e3ebe10573a4863b3734daf8f3c4382"
format_stamp: "Formatted at 2024-05-01 14:08:49 on dist-test-slave-bkzb"
I20240501 14:08:55.802544 26981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:55.813552 26981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:55.814077 26981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:55.814214 26981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:55.814693 26981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:55.815511 26981 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:55.815558 26981 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:55.815610 26981 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:55.815637 26981 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:55.824965 26981 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.134:42583
I20240501 14:08:55.824986 27108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.134:42583 every 8 connection(s)
I20240501 14:08:55.825915 26981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:08:55.826061 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 26981
I20240501 14:08:55.826494 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.135:39631
--local_ip_for_outbound_sockets=127.22.252.135
--tserver_master_addrs=127.22.252.190:44241
--webserver_port=40343
--webserver_interface=127.22.252.135
--builtin_ntp_servers=127.22.252.148:35091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:08:55.836328 27109 heartbeater.cc:347] Connected to a master server at 127.22.252.190:44241
I20240501 14:08:55.836568 27109 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:55.837070 27109 heartbeater.cc:511] Master 127.22.252.190:44241 requested a full tablet report, sending...
I20240501 14:08:55.838109 25387 ts_manager.cc:194] Re-registered known tserver with Master: 0e3ebe10573a4863b3734daf8f3c4382 (127.22.252.134:42583)
I20240501 14:08:55.838634 25387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.134:46047
W20240501 14:08:55.933027 27111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:55.933221 27111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:55.933266 27111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:55.936445 27111 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:08:55.936506 27111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:55.936596 27111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.135
I20240501 14:08:55.940135 27111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:35091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.135:39631
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.252.135
--webserver_port=40343
--tserver_master_addrs=127.22.252.190:44241
--never_fsync=true
--heap_profile_path=/tmp/kudu.27111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.135
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:55.940968 27111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:55.942032 27111 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:55.950145 27120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:55.951107 27118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:55.951886 27117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:55.952701 27111 server_base.cc:1034] running on GCE node
I20240501 14:08:55.953016 27111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:55.953562 27111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:55.954725 27111 hybrid_clock.cc:648] HybridClock initialized: now 1714572535954703 us; error 31 us; skew 500 ppm
I20240501 14:08:55.956463 27111 webserver.cc:460] Webserver started at http://127.22.252.135:40343/ using document root <none> and password file <none>
I20240501 14:08:55.956974 27111 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:55.957053 27111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:55.960191 27111 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:08:55.961923 27126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:55.962826 27111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240501 14:08:55.962926 27111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "3df2596a705349f185baa3b69dabfc25"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-bkzb"
I20240501 14:08:55.963289 27111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:55.977392 27111 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:55.977911 27111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:55.978055 27111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:55.978631 27111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:55.979509 27111 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:55.979562 27111 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:55.979621 27111 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:55.979653 27111 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:55.988406 27111 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.135:39631
I20240501 14:08:55.988468 27239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.135:39631 every 8 connection(s)
I20240501 14:08:55.989347 27111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:08:55.992457 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 27111
I20240501 14:08:55.998085 27240 heartbeater.cc:347] Connected to a master server at 127.22.252.190:44241
I20240501 14:08:55.998293 27240 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:55.998687 27240 heartbeater.cc:511] Master 127.22.252.190:44241 requested a full tablet report, sending...
I20240501 14:08:55.999684 25387 ts_manager.cc:194] Re-registered known tserver with Master: 3df2596a705349f185baa3b69dabfc25 (127.22.252.135:39631)
I20240501 14:08:56.000517 25387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.135:54065
I20240501 14:08:56.180722 27041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.181188 26772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:56.183557 25509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:56.190151 26640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.191776 26912 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.193226 27174 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.207446 25640 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:56.309980 26912 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.315112 26640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.316022 25640 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:56.316744 27041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.320714 26706 heartbeater.cc:503] Master 127.22.252.190:44241 was elected leader, sending a full tablet report...
I20240501 14:08:56.325253 25509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:56.336987 27174 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.342834 26772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:56.380136 25660 consensus_queue.cc:237] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } }
W20240501 14:08:56.385046 25594 consensus_peers.cc:485] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d -> Peer e9e15d9915264a62a5c9e68f67872199 (127.22.252.133:42377): Couldn't send request to peer e9e15d9915264a62a5c9e68f67872199. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd9a83b1cc1441da972fb41466aaf402. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:56.677843 26978 heartbeater.cc:503] Master 127.22.252.190:44241 was elected leader, sending a full tablet report...
I20240501 14:08:56.840795 27109 heartbeater.cc:503] Master 127.22.252.190:44241 was elected leader, sending a full tablet report...
I20240501 14:08:56.890914 27343 ts_tablet_manager.cc:902] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199: Initiating tablet copy from peer c6717727de344c0b90dd50fd5e70883d (127.22.252.130:44715)
I20240501 14:08:56.891656 27343 tablet_copy_client.cc:308] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.130:44715
I20240501 14:08:56.895007 25680 tablet_copy_service.cc:140] P c6717727de344c0b90dd50fd5e70883d: Received BeginTabletCopySession request for tablet fd9a83b1cc1441da972fb41466aaf402 from peer e9e15d9915264a62a5c9e68f67872199 ({username='slave'} at 127.22.252.133:39233)
I20240501 14:08:56.895138 25680 tablet_copy_service.cc:161] P c6717727de344c0b90dd50fd5e70883d: Beginning new tablet copy session on tablet fd9a83b1cc1441da972fb41466aaf402 from peer e9e15d9915264a62a5c9e68f67872199 at {username='slave'} at 127.22.252.133:39233: session id = e9e15d9915264a62a5c9e68f67872199-fd9a83b1cc1441da972fb41466aaf402
I20240501 14:08:56.896335 25680 tablet_copy_source_session.cc:205] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:08:56.899433 27343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd9a83b1cc1441da972fb41466aaf402. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:56.904423 27343 tablet_copy_client.cc:783] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199: tablet copy: Starting download of 0 data blocks...
I20240501 14:08:56.904567 27343 tablet_copy_client.cc:647] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199: tablet copy: Starting download of 1 WAL segments...
I20240501 14:08:56.906116 27343 tablet_copy_client.cc:523] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:08:56.907583 27343 tablet_bootstrap.cc:492] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199: Bootstrap starting.
I20240501 14:08:56.912458 27343 log.cc:824] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:56.930801 27343 tablet_bootstrap.cc:492] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:08:56.931170 27343 tablet_bootstrap.cc:492] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199: Bootstrap complete.
I20240501 14:08:56.931437 27343 ts_tablet_manager.cc:1367] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199: Time spent bootstrapping tablet: real 0.024s user 0.023s sys 0.000s
I20240501 14:08:56.933315 27343 raft_consensus.cc:348] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:56.933826 27343 raft_consensus.cc:729] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9e15d9915264a62a5c9e68f67872199, State: Initialized, Role: FOLLOWER
I20240501 14:08:56.934108 27343 consensus_queue.cc:260] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } }
I20240501 14:08:56.934990 27343 ts_tablet_manager.cc:1398] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240501 14:08:56.935773 25680 tablet_copy_service.cc:342] P c6717727de344c0b90dd50fd5e70883d: Request end of tablet copy session e9e15d9915264a62a5c9e68f67872199-fd9a83b1cc1441da972fb41466aaf402 received from {username='slave'} at 127.22.252.133:39233
I20240501 14:08:56.935860 25680 tablet_copy_service.cc:433] P c6717727de344c0b90dd50fd5e70883d: ending tablet copy session e9e15d9915264a62a5c9e68f67872199-fd9a83b1cc1441da972fb41466aaf402 on tablet fd9a83b1cc1441da972fb41466aaf402 with peer e9e15d9915264a62a5c9e68f67872199
I20240501 14:08:57.002368 27240 heartbeater.cc:503] Master 127.22.252.190:44241 was elected leader, sending a full tablet report...
I20240501 14:08:57.454660 26932 raft_consensus.cc:1203] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:08:57.456034 27354 raft_consensus.cc:2932] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e9e15d9915264a62a5c9e68f67872199 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } } }
I20240501 14:08:57.456557 26932 raft_consensus.cc:2932] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e9e15d9915264a62a5c9e68f67872199 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } } }
I20240501 14:08:57.457602 27354 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:60554 (request call id 0) took 1078 ms. Trace:
I20240501 14:08:57.457702 27354 rpcz_store.cc:276] 0501 14:08:56.379572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 14:08:56.379619 (+ 47us) service_pool.cc:225] Handling call
0501 14:08:56.380923 (+ 1304us) log.cc:842] Serialized 263 byte log entry
0501 14:08:57.457589 (+1076666us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":42,"raft.queue_time_us":91,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":287,"spinlock_wait_cycles":14208,"thread_start_us":90,"threads_started":2,"wal-append.queue_time_us":119,"wal-append.run_cpu_time_us":294,"wal-append.run_wall_time_us":1000361}
I20240501 14:08:57.458850 25386 catalog_manager.cc:5484] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199 reported cstate change: config changed from index -1 to 4, VOTER e9e15d9915264a62a5c9e68f67872199 (127.22.252.133) added. New cstate: current_term: 1 leader_uuid: "c6717727de344c0b90dd50fd5e70883d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } } }
I20240501 14:08:57.477766 25529 consensus_queue.cc:237] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } }
W20240501 14:08:57.482811 25462 consensus_peers.cc:485] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 -> Peer 8a9c183caefa4437b53752c1203b8bb7 (127.22.252.131:36947): Couldn't send request to peer 8a9c183caefa4437b53752c1203b8bb7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d16f3d9f72b44c2cb15a6c634bee2a28. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:57.997287 27383 ts_tablet_manager.cc:902] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7: Initiating tablet copy from peer e8f33be94f864561bec2e185b5a695c3 (127.22.252.129:44353)
I20240501 14:08:57.998028 27383 tablet_copy_client.cc:308] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.129:44353
I20240501 14:08:58.001695 25549 tablet_copy_service.cc:140] P e8f33be94f864561bec2e185b5a695c3: Received BeginTabletCopySession request for tablet d16f3d9f72b44c2cb15a6c634bee2a28 from peer 8a9c183caefa4437b53752c1203b8bb7 ({username='slave'} at 127.22.252.131:35661)
I20240501 14:08:58.001821 25549 tablet_copy_service.cc:161] P e8f33be94f864561bec2e185b5a695c3: Beginning new tablet copy session on tablet d16f3d9f72b44c2cb15a6c634bee2a28 from peer 8a9c183caefa4437b53752c1203b8bb7 at {username='slave'} at 127.22.252.131:35661: session id = 8a9c183caefa4437b53752c1203b8bb7-d16f3d9f72b44c2cb15a6c634bee2a28
I20240501 14:08:58.002912 25549 tablet_copy_source_session.cc:205] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:08:58.006285 27383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d16f3d9f72b44c2cb15a6c634bee2a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:58.011301 27383 tablet_copy_client.cc:783] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7: tablet copy: Starting download of 0 data blocks...
I20240501 14:08:58.011502 27383 tablet_copy_client.cc:647] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:08:58.013113 27383 tablet_copy_client.cc:523] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:08:58.014525 27383 tablet_bootstrap.cc:492] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7: Bootstrap starting.
I20240501 14:08:58.019539 27383 log.cc:824] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:58.038861 27383 tablet_bootstrap.cc:492] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:08:58.039203 27383 tablet_bootstrap.cc:492] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7: Bootstrap complete.
I20240501 14:08:58.039474 27383 ts_tablet_manager.cc:1367] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.000s
I20240501 14:08:58.041363 27383 raft_consensus.cc:348] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:08:58.041903 27383 raft_consensus.cc:729] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c183caefa4437b53752c1203b8bb7, State: Initialized, Role: FOLLOWER
I20240501 14:08:58.042213 27383 consensus_queue.cc:260] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } }
I20240501 14:08:58.042999 27383 ts_tablet_manager.cc:1398] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240501 14:08:58.043862 25549 tablet_copy_service.cc:342] P e8f33be94f864561bec2e185b5a695c3: Request end of tablet copy session 8a9c183caefa4437b53752c1203b8bb7-d16f3d9f72b44c2cb15a6c634bee2a28 received from {username='slave'} at 127.22.252.131:35661
I20240501 14:08:58.043957 25549 tablet_copy_service.cc:433] P e8f33be94f864561bec2e185b5a695c3: ending tablet copy session 8a9c183caefa4437b53752c1203b8bb7-d16f3d9f72b44c2cb15a6c634bee2a28 on tablet d16f3d9f72b44c2cb15a6c634bee2a28 with peer 8a9c183caefa4437b53752c1203b8bb7
I20240501 14:08:58.557601 26660 raft_consensus.cc:1203] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:08:58.559053 27393 raft_consensus.cc:2932] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a9c183caefa4437b53752c1203b8bb7 (127.22.252.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } } }
I20240501 14:08:58.559547 26660 raft_consensus.cc:2932] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a9c183caefa4437b53752c1203b8bb7 (127.22.252.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } } }
I20240501 14:08:58.560588 27393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:44876 (request call id 0) took 1083 ms. Trace:
I20240501 14:08:58.560669 27393 rpcz_store.cc:276] 0501 14:08:57.477207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 14:08:57.477258 (+ 51us) service_pool.cc:225] Handling call
0501 14:08:57.478588 (+ 1330us) log.cc:842] Serialized 263 byte log entry
0501 14:08:58.560571 (+1081983us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":48,"raft.queue_time_us":109,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":276,"spinlock_wait_cycles":59776,"thread_start_us":110,"threads_started":2,"wal-append.queue_time_us":135,"wal-append.run_cpu_time_us":264,"wal-append.run_wall_time_us":1000329}
I20240501 14:08:58.563163 25386 catalog_manager.cc:5484] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 reported cstate change: config changed from index -1 to 4, VOTER 8a9c183caefa4437b53752c1203b8bb7 (127.22.252.131) added. New cstate: current_term: 1 leader_uuid: "e8f33be94f864561bec2e185b5a695c3" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } } }
I20240501 14:08:58.579653 25660 consensus_queue.cc:237] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } }
W20240501 14:08:58.584170 25595 consensus_peers.cc:485] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d -> Peer 0e3ebe10573a4863b3734daf8f3c4382 (127.22.252.134:42583): Couldn't send request to peer 0e3ebe10573a4863b3734daf8f3c4382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a58883c2a8074e01b2170310e3008e52. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:59.188797 27422 ts_tablet_manager.cc:902] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382: Initiating tablet copy from peer c6717727de344c0b90dd50fd5e70883d (127.22.252.130:44715)
I20240501 14:08:59.189519 27422 tablet_copy_client.cc:308] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.130:44715
I20240501 14:08:59.193066 25680 tablet_copy_service.cc:140] P c6717727de344c0b90dd50fd5e70883d: Received BeginTabletCopySession request for tablet a58883c2a8074e01b2170310e3008e52 from peer 0e3ebe10573a4863b3734daf8f3c4382 ({username='slave'} at 127.22.252.134:56181)
I20240501 14:08:59.193171 25680 tablet_copy_service.cc:161] P c6717727de344c0b90dd50fd5e70883d: Beginning new tablet copy session on tablet a58883c2a8074e01b2170310e3008e52 from peer 0e3ebe10573a4863b3734daf8f3c4382 at {username='slave'} at 127.22.252.134:56181: session id = 0e3ebe10573a4863b3734daf8f3c4382-a58883c2a8074e01b2170310e3008e52
I20240501 14:08:59.193904 25680 tablet_copy_source_session.cc:205] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:08:59.197300 27422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a58883c2a8074e01b2170310e3008e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.202967 27422 tablet_copy_client.cc:783] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382: tablet copy: Starting download of 0 data blocks...
I20240501 14:08:59.203159 27422 tablet_copy_client.cc:647] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382: tablet copy: Starting download of 1 WAL segments...
I20240501 14:08:59.204741 27422 tablet_copy_client.cc:523] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:08:59.232165 27422 tablet_bootstrap.cc:492] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382: Bootstrap starting.
I20240501 14:08:59.237967 27422 log.cc:824] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:59.257504 27422 tablet_bootstrap.cc:492] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:08:59.257947 27422 tablet_bootstrap.cc:492] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382: Bootstrap complete.
I20240501 14:08:59.258291 27422 ts_tablet_manager.cc:1367] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382: Time spent bootstrapping tablet: real 0.026s user 0.021s sys 0.004s
I20240501 14:08:59.260315 27422 raft_consensus.cc:348] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:08:59.260892 27422 raft_consensus.cc:729] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e3ebe10573a4863b3734daf8f3c4382, State: Initialized, Role: FOLLOWER
I20240501 14:08:59.261201 27422 consensus_queue.cc:260] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } }
I20240501 14:08:59.262058 27422 ts_tablet_manager.cc:1398] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 14:08:59.262827 25680 tablet_copy_service.cc:342] P c6717727de344c0b90dd50fd5e70883d: Request end of tablet copy session 0e3ebe10573a4863b3734daf8f3c4382-a58883c2a8074e01b2170310e3008e52 received from {username='slave'} at 127.22.252.134:56181
I20240501 14:08:59.262919 25680 tablet_copy_service.cc:433] P c6717727de344c0b90dd50fd5e70883d: ending tablet copy session 0e3ebe10573a4863b3734daf8f3c4382-a58883c2a8074e01b2170310e3008e52 on tablet a58883c2a8074e01b2170310e3008e52 with peer 0e3ebe10573a4863b3734daf8f3c4382
I20240501 14:08:59.662621 27063 raft_consensus.cc:1203] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:08:59.663946 27432 raft_consensus.cc:2932] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0e3ebe10573a4863b3734daf8f3c4382 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } } }
I20240501 14:08:59.664467 27063 raft_consensus.cc:2932] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0e3ebe10573a4863b3734daf8f3c4382 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } } }
I20240501 14:08:59.665642 27432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:59110 (request call id 0) took 1086 ms. Trace:
I20240501 14:08:59.665786 27432 rpcz_store.cc:276] 0501 14:08:58.579305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 14:08:58.579361 (+ 56us) service_pool.cc:225] Handling call
0501 14:08:58.580090 (+ 729us) log.cc:842] Serialized 263 byte log entry
0501 14:08:59.665625 (+1085535us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":41,"raft.queue_time_us":54,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":112,"thread_start_us":60,"threads_started":1,"wal-append.queue_time_us":145,"wal-append.run_cpu_time_us":310,"wal-append.run_wall_time_us":1000371}
I20240501 14:08:59.666978 25378 catalog_manager.cc:5484] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382 reported cstate change: config changed from index -1 to 4, VOTER 0e3ebe10573a4863b3734daf8f3c4382 (127.22.252.134) added. New cstate: current_term: 1 leader_uuid: "c6717727de344c0b90dd50fd5e70883d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } } }
I20240501 14:08:59.688177 25529 consensus_queue.cc:237] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } }
W20240501 14:08:59.692593 25465 consensus_peers.cc:485] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 -> Peer 3df2596a705349f185baa3b69dabfc25 (127.22.252.135:39631): Couldn't send request to peer 3df2596a705349f185baa3b69dabfc25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36b4476375e546298f3d8184dda64204. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:00.145237 27459 ts_tablet_manager.cc:902] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25: Initiating tablet copy from peer e8f33be94f864561bec2e185b5a695c3 (127.22.252.129:44353)
I20240501 14:09:00.145969 27459 tablet_copy_client.cc:308] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.129:44353
I20240501 14:09:00.149510 25549 tablet_copy_service.cc:140] P e8f33be94f864561bec2e185b5a695c3: Received BeginTabletCopySession request for tablet 36b4476375e546298f3d8184dda64204 from peer 3df2596a705349f185baa3b69dabfc25 ({username='slave'} at 127.22.252.135:56757)
I20240501 14:09:00.149606 25549 tablet_copy_service.cc:161] P e8f33be94f864561bec2e185b5a695c3: Beginning new tablet copy session on tablet 36b4476375e546298f3d8184dda64204 from peer 3df2596a705349f185baa3b69dabfc25 at {username='slave'} at 127.22.252.135:56757: session id = 3df2596a705349f185baa3b69dabfc25-36b4476375e546298f3d8184dda64204
I20240501 14:09:00.150393 25549 tablet_copy_source_session.cc:205] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:00.153635 27459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36b4476375e546298f3d8184dda64204. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:00.159021 27459 tablet_copy_client.cc:783] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:00.159207 27459 tablet_copy_client.cc:647] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:00.160708 27459 tablet_copy_client.cc:523] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:00.162462 27459 tablet_bootstrap.cc:492] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25: Bootstrap starting.
I20240501 14:09:00.167290 27459 log.cc:824] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:00.186290 27459 tablet_bootstrap.cc:492] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:09:00.186631 27459 tablet_bootstrap.cc:492] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25: Bootstrap complete.
I20240501 14:09:00.186904 27459 ts_tablet_manager.cc:1367] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.000s
I20240501 14:09:00.188793 27459 raft_consensus.cc:348] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:09:00.189316 27459 raft_consensus.cc:729] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3df2596a705349f185baa3b69dabfc25, State: Initialized, Role: FOLLOWER
I20240501 14:09:00.189603 27459 consensus_queue.cc:260] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } }
I20240501 14:09:00.190310 27459 ts_tablet_manager.cc:1398] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240501 14:09:00.191064 25549 tablet_copy_service.cc:342] P e8f33be94f864561bec2e185b5a695c3: Request end of tablet copy session 3df2596a705349f185baa3b69dabfc25-36b4476375e546298f3d8184dda64204 received from {username='slave'} at 127.22.252.135:56757
I20240501 14:09:00.191157 25549 tablet_copy_service.cc:433] P e8f33be94f864561bec2e185b5a695c3: ending tablet copy session 3df2596a705349f185baa3b69dabfc25-36b4476375e546298f3d8184dda64204 on tablet 36b4476375e546298f3d8184dda64204 with peer 3df2596a705349f185baa3b69dabfc25
I20240501 14:09:00.736958 27194 raft_consensus.cc:1203] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:00.738309 27469 raft_consensus.cc:2932] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3df2596a705349f185baa3b69dabfc25 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } } }
I20240501 14:09:00.738826 27194 raft_consensus.cc:2932] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3df2596a705349f185baa3b69dabfc25 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } } }
I20240501 14:09:00.739838 27469 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35884 (request call id 0) took 1052 ms. Trace:
I20240501 14:09:00.739959 27469 rpcz_store.cc:276] 0501 14:08:59.687852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 14:08:59.687907 (+ 55us) service_pool.cc:225] Handling call
0501 14:08:59.688638 (+ 731us) log.cc:842] Serialized 263 byte log entry
0501 14:09:00.739818 (+1051180us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":38,"raft.queue_time_us":51,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":112,"thread_start_us":62,"threads_started":1,"wal-append.queue_time_us":127,"wal-append.run_cpu_time_us":335,"wal-append.run_wall_time_us":1000401}
I20240501 14:09:00.741865 25386 catalog_manager.cc:5484] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25 reported cstate change: config changed from index -1 to 4, VOTER 3df2596a705349f185baa3b69dabfc25 (127.22.252.135) added. New cstate: current_term: 1 leader_uuid: "e8f33be94f864561bec2e185b5a695c3" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } } }
I20240501 14:09:00.758469 25529 consensus_queue.cc:237] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "86b991e2c8414b6698e51851d34bc8f2" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 37205 } }
W20240501 14:09:00.763100 25462 consensus_peers.cc:485] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 -> Peer 86b991e2c8414b6698e51851d34bc8f2 (127.22.252.132:37205): Couldn't send request to peer 86b991e2c8414b6698e51851d34bc8f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a1f7cc1afb143a8a132944682f969fe. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:01.343806 27496 ts_tablet_manager.cc:902] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2: Initiating tablet copy from peer e8f33be94f864561bec2e185b5a695c3 (127.22.252.129:44353)
I20240501 14:09:01.344530 27496 tablet_copy_client.cc:308] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.129:44353
I20240501 14:09:01.347908 25549 tablet_copy_service.cc:140] P e8f33be94f864561bec2e185b5a695c3: Received BeginTabletCopySession request for tablet 8a1f7cc1afb143a8a132944682f969fe from peer 86b991e2c8414b6698e51851d34bc8f2 ({username='slave'} at 127.22.252.132:41109)
I20240501 14:09:01.348031 25549 tablet_copy_service.cc:161] P e8f33be94f864561bec2e185b5a695c3: Beginning new tablet copy session on tablet 8a1f7cc1afb143a8a132944682f969fe from peer 86b991e2c8414b6698e51851d34bc8f2 at {username='slave'} at 127.22.252.132:41109: session id = 86b991e2c8414b6698e51851d34bc8f2-8a1f7cc1afb143a8a132944682f969fe
I20240501 14:09:01.348917 25549 tablet_copy_source_session.cc:205] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:01.350512 27496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a1f7cc1afb143a8a132944682f969fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:01.353070 27496 tablet_copy_client.cc:783] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:01.353282 27496 tablet_copy_client.cc:647] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:01.354597 27496 tablet_copy_client.cc:523] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:01.355666 27496 tablet_bootstrap.cc:492] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2: Bootstrap starting.
I20240501 14:09:01.373719 27496 tablet_bootstrap.cc:492] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:09:01.374066 27496 tablet_bootstrap.cc:492] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2: Bootstrap complete.
I20240501 14:09:01.374198 27496 ts_tablet_manager.cc:1367] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2: Time spent bootstrapping tablet: real 0.019s user 0.018s sys 0.000s
I20240501 14:09:01.374581 27496 raft_consensus.cc:348] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:09:01.374819 27496 raft_consensus.cc:729] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b991e2c8414b6698e51851d34bc8f2, State: Initialized, Role: FOLLOWER
I20240501 14:09:01.374912 27496 consensus_queue.cc:260] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "86b991e2c8414b6698e51851d34bc8f2" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 37205 } }
I20240501 14:09:01.375233 27496 ts_tablet_manager.cc:1398] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 14:09:01.375790 25549 tablet_copy_service.cc:342] P e8f33be94f864561bec2e185b5a695c3: Request end of tablet copy session 86b991e2c8414b6698e51851d34bc8f2-8a1f7cc1afb143a8a132944682f969fe received from {username='slave'} at 127.22.252.132:41109
I20240501 14:09:01.375870 25549 tablet_copy_service.cc:433] P e8f33be94f864561bec2e185b5a695c3: ending tablet copy session 86b991e2c8414b6698e51851d34bc8f2-8a1f7cc1afb143a8a132944682f969fe on tablet 8a1f7cc1afb143a8a132944682f969fe with peer 86b991e2c8414b6698e51851d34bc8f2
I20240501 14:09:01.823668 26792 raft_consensus.cc:1203] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:01.824750 27505 raft_consensus.cc:2932] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 86b991e2c8414b6698e51851d34bc8f2 (127.22.252.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "86b991e2c8414b6698e51851d34bc8f2" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 37205 } } }
I20240501 14:09:01.825210 26792 raft_consensus.cc:2932] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 86b991e2c8414b6698e51851d34bc8f2 (127.22.252.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "86b991e2c8414b6698e51851d34bc8f2" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 37205 } } }
I20240501 14:09:01.826217 27505 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35892 (request call id 0) took 1068 ms. Trace:
I20240501 14:09:01.826339 27505 rpcz_store.cc:276] 0501 14:09:00.758143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 14:09:00.758207 (+ 64us) service_pool.cc:225] Handling call
0501 14:09:00.758858 (+ 651us) log.cc:842] Serialized 263 byte log entry
0501 14:09:01.826199 (+1067341us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":38,"raft.queue_time_us":33,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":131,"spinlock_wait_cycles":11008,"thread_start_us":86,"threads_started":1,"wal-append.queue_time_us":159,"wal-append.run_cpu_time_us":307,"wal-append.run_wall_time_us":1000373}
I20240501 14:09:01.827886 25378 catalog_manager.cc:5484] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 reported cstate change: config changed from index -1 to 4, VOTER 86b991e2c8414b6698e51851d34bc8f2 (127.22.252.132) added. New cstate: current_term: 1 leader_uuid: "e8f33be94f864561bec2e185b5a695c3" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "86b991e2c8414b6698e51851d34bc8f2" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 37205 } } }
I20240501 14:09:01.844859 25529 consensus_queue.cc:237] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } }
W20240501 14:09:01.848707 25463 consensus_peers.cc:485] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 -> Peer e9e15d9915264a62a5c9e68f67872199 (127.22.252.133:42377): Couldn't send request to peer e9e15d9915264a62a5c9e68f67872199. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c230502585ed4dd1aa75bc02a3e7d242. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:02.341430 27531 ts_tablet_manager.cc:902] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199: Initiating tablet copy from peer e8f33be94f864561bec2e185b5a695c3 (127.22.252.129:44353)
I20240501 14:09:02.341941 27531 tablet_copy_client.cc:308] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.129:44353
I20240501 14:09:02.345450 25549 tablet_copy_service.cc:140] P e8f33be94f864561bec2e185b5a695c3: Received BeginTabletCopySession request for tablet c230502585ed4dd1aa75bc02a3e7d242 from peer e9e15d9915264a62a5c9e68f67872199 ({username='slave'} at 127.22.252.133:48659)
I20240501 14:09:02.345571 25549 tablet_copy_service.cc:161] P e8f33be94f864561bec2e185b5a695c3: Beginning new tablet copy session on tablet c230502585ed4dd1aa75bc02a3e7d242 from peer e9e15d9915264a62a5c9e68f67872199 at {username='slave'} at 127.22.252.133:48659: session id = e9e15d9915264a62a5c9e68f67872199-c230502585ed4dd1aa75bc02a3e7d242
I20240501 14:09:02.346366 25549 tablet_copy_source_session.cc:205] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:02.347188 27531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c230502585ed4dd1aa75bc02a3e7d242. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:02.349042 27531 tablet_copy_client.cc:783] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:02.349182 27531 tablet_copy_client.cc:647] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:02.350178 27531 tablet_copy_client.cc:523] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:02.351084 27531 tablet_bootstrap.cc:492] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199: Bootstrap starting.
I20240501 14:09:02.368095 27531 tablet_bootstrap.cc:492] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:09:02.368362 27531 tablet_bootstrap.cc:492] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199: Bootstrap complete.
I20240501 14:09:02.368512 27531 ts_tablet_manager.cc:1367] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.007s
I20240501 14:09:02.369012 27531 raft_consensus.cc:348] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:09:02.369215 27531 raft_consensus.cc:729] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9e15d9915264a62a5c9e68f67872199, State: Initialized, Role: FOLLOWER
I20240501 14:09:02.369307 27531 consensus_queue.cc:260] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } }
I20240501 14:09:02.369629 27531 ts_tablet_manager.cc:1398] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:02.370198 25549 tablet_copy_service.cc:342] P e8f33be94f864561bec2e185b5a695c3: Request end of tablet copy session e9e15d9915264a62a5c9e68f67872199-c230502585ed4dd1aa75bc02a3e7d242 received from {username='slave'} at 127.22.252.133:48659
I20240501 14:09:02.370281 25549 tablet_copy_service.cc:433] P e8f33be94f864561bec2e185b5a695c3: ending tablet copy session e9e15d9915264a62a5c9e68f67872199-c230502585ed4dd1aa75bc02a3e7d242 on tablet c230502585ed4dd1aa75bc02a3e7d242 with peer e9e15d9915264a62a5c9e68f67872199
I20240501 14:09:02.778861 26932 raft_consensus.cc:1203] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:02.779827 27539 raft_consensus.cc:2932] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e9e15d9915264a62a5c9e68f67872199 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } } }
I20240501 14:09:02.780340 26932 raft_consensus.cc:2932] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e9e15d9915264a62a5c9e68f67872199 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } } }
I20240501 14:09:02.782616 25386 catalog_manager.cc:5484] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 reported cstate change: config changed from index -1 to 4, VOTER e9e15d9915264a62a5c9e68f67872199 (127.22.252.133) added. New cstate: current_term: 1 leader_uuid: "e8f33be94f864561bec2e185b5a695c3" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } } }
I20240501 14:09:02.798658 25660 consensus_queue.cc:237] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } }
W20240501 14:09:02.802484 25593 consensus_peers.cc:485] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d -> Peer 8a9c183caefa4437b53752c1203b8bb7 (127.22.252.131:36947): Couldn't send request to peer 8a9c183caefa4437b53752c1203b8bb7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e9029b14eda48cab26e246ed7535d65. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:03.287649 27564 ts_tablet_manager.cc:902] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7: Initiating tablet copy from peer c6717727de344c0b90dd50fd5e70883d (127.22.252.130:44715)
I20240501 14:09:03.288161 27564 tablet_copy_client.cc:308] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.130:44715
I20240501 14:09:03.291679 25680 tablet_copy_service.cc:140] P c6717727de344c0b90dd50fd5e70883d: Received BeginTabletCopySession request for tablet 4e9029b14eda48cab26e246ed7535d65 from peer 8a9c183caefa4437b53752c1203b8bb7 ({username='slave'} at 127.22.252.131:48217)
I20240501 14:09:03.291808 25680 tablet_copy_service.cc:161] P c6717727de344c0b90dd50fd5e70883d: Beginning new tablet copy session on tablet 4e9029b14eda48cab26e246ed7535d65 from peer 8a9c183caefa4437b53752c1203b8bb7 at {username='slave'} at 127.22.252.131:48217: session id = 8a9c183caefa4437b53752c1203b8bb7-4e9029b14eda48cab26e246ed7535d65
I20240501 14:09:03.292646 25680 tablet_copy_source_session.cc:205] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:03.293498 27564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e9029b14eda48cab26e246ed7535d65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:03.295212 27564 tablet_copy_client.cc:783] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:03.295368 27564 tablet_copy_client.cc:647] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:03.296496 27564 tablet_copy_client.cc:523] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:03.297453 27564 tablet_bootstrap.cc:492] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7: Bootstrap starting.
I20240501 14:09:03.315078 27564 tablet_bootstrap.cc:492] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:09:03.315352 27564 tablet_bootstrap.cc:492] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7: Bootstrap complete.
I20240501 14:09:03.315506 27564 ts_tablet_manager.cc:1367] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20240501 14:09:03.315966 27564 raft_consensus.cc:348] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:09:03.316182 27564 raft_consensus.cc:729] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9c183caefa4437b53752c1203b8bb7, State: Initialized, Role: FOLLOWER
I20240501 14:09:03.316295 27564 consensus_queue.cc:260] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } }
I20240501 14:09:03.316553 27564 ts_tablet_manager.cc:1398] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20240501 14:09:03.317279 25680 tablet_copy_service.cc:342] P c6717727de344c0b90dd50fd5e70883d: Request end of tablet copy session 8a9c183caefa4437b53752c1203b8bb7-4e9029b14eda48cab26e246ed7535d65 received from {username='slave'} at 127.22.252.131:48217
I20240501 14:09:03.317359 25680 tablet_copy_service.cc:433] P c6717727de344c0b90dd50fd5e70883d: ending tablet copy session 8a9c183caefa4437b53752c1203b8bb7-4e9029b14eda48cab26e246ed7535d65 on tablet 4e9029b14eda48cab26e246ed7535d65 with peer 8a9c183caefa4437b53752c1203b8bb7
I20240501 14:09:03.877774 26660 raft_consensus.cc:1203] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:03.878774 27573 raft_consensus.cc:2932] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a9c183caefa4437b53752c1203b8bb7 (127.22.252.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } } }
I20240501 14:09:03.879226 26660 raft_consensus.cc:2932] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a9c183caefa4437b53752c1203b8bb7 (127.22.252.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } } }
I20240501 14:09:03.880328 27573 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:59136 (request call id 0) took 1082 ms. Trace:
I20240501 14:09:03.880450 27573 rpcz_store.cc:276] 0501 14:09:02.798351 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 14:09:02.798405 (+ 54us) service_pool.cc:225] Handling call
0501 14:09:02.799034 (+ 629us) log.cc:842] Serialized 263 byte log entry
0501 14:09:03.880308 (+1081274us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":36,"raft.queue_time_us":44,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":106,"thread_start_us":59,"threads_started":1,"wal-append.queue_time_us":122,"wal-append.run_cpu_time_us":296,"wal-append.run_wall_time_us":1000358}
I20240501 14:09:03.881742 25378 catalog_manager.cc:5484] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7 reported cstate change: config changed from index -1 to 4, VOTER 8a9c183caefa4437b53752c1203b8bb7 (127.22.252.131) added. New cstate: current_term: 1 leader_uuid: "c6717727de344c0b90dd50fd5e70883d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } } }
I20240501 14:09:03.899295 25529 consensus_queue.cc:237] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } }
W20240501 14:09:03.903235 25464 consensus_peers.cc:485] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 -> Peer 0e3ebe10573a4863b3734daf8f3c4382 (127.22.252.134:42583): Couldn't send request to peer 0e3ebe10573a4863b3734daf8f3c4382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78ff5bcb4c5445659d365e84dc810ebc. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:04.283103 27600 ts_tablet_manager.cc:902] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382: Initiating tablet copy from peer e8f33be94f864561bec2e185b5a695c3 (127.22.252.129:44353)
I20240501 14:09:04.283586 27600 tablet_copy_client.cc:308] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.129:44353
I20240501 14:09:04.287217 25549 tablet_copy_service.cc:140] P e8f33be94f864561bec2e185b5a695c3: Received BeginTabletCopySession request for tablet 78ff5bcb4c5445659d365e84dc810ebc from peer 0e3ebe10573a4863b3734daf8f3c4382 ({username='slave'} at 127.22.252.134:45599)
I20240501 14:09:04.287317 25549 tablet_copy_service.cc:161] P e8f33be94f864561bec2e185b5a695c3: Beginning new tablet copy session on tablet 78ff5bcb4c5445659d365e84dc810ebc from peer 0e3ebe10573a4863b3734daf8f3c4382 at {username='slave'} at 127.22.252.134:45599: session id = 0e3ebe10573a4863b3734daf8f3c4382-78ff5bcb4c5445659d365e84dc810ebc
I20240501 14:09:04.288075 25549 tablet_copy_source_session.cc:205] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:04.289126 27600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ff5bcb4c5445659d365e84dc810ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:04.290771 27600 tablet_copy_client.cc:783] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:04.290930 27600 tablet_copy_client.cc:647] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:04.291956 27600 tablet_copy_client.cc:523] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:04.292953 27600 tablet_bootstrap.cc:492] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382: Bootstrap starting.
I20240501 14:09:04.311483 27600 tablet_bootstrap.cc:492] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:09:04.311770 27600 tablet_bootstrap.cc:492] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382: Bootstrap complete.
I20240501 14:09:04.311918 27600 ts_tablet_manager.cc:1367] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.000s
I20240501 14:09:04.312515 27600 raft_consensus.cc:348] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:09:04.312736 27600 raft_consensus.cc:729] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e3ebe10573a4863b3734daf8f3c4382, State: Initialized, Role: FOLLOWER
I20240501 14:09:04.312832 27600 consensus_queue.cc:260] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } }
I20240501 14:09:04.313221 27600 ts_tablet_manager.cc:1398] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240501 14:09:04.313854 25549 tablet_copy_service.cc:342] P e8f33be94f864561bec2e185b5a695c3: Request end of tablet copy session 0e3ebe10573a4863b3734daf8f3c4382-78ff5bcb4c5445659d365e84dc810ebc received from {username='slave'} at 127.22.252.134:45599
I20240501 14:09:04.313957 25549 tablet_copy_service.cc:433] P e8f33be94f864561bec2e185b5a695c3: ending tablet copy session 0e3ebe10573a4863b3734daf8f3c4382-78ff5bcb4c5445659d365e84dc810ebc on tablet 78ff5bcb4c5445659d365e84dc810ebc with peer 0e3ebe10573a4863b3734daf8f3c4382
I20240501 14:09:04.905593 27063 raft_consensus.cc:1203] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:04.906497 27609 raft_consensus.cc:2932] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0e3ebe10573a4863b3734daf8f3c4382 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } } }
I20240501 14:09:04.907003 27063 raft_consensus.cc:2932] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0e3ebe10573a4863b3734daf8f3c4382 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } } }
I20240501 14:09:04.908087 27609 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35936 (request call id 0) took 1009 ms. Trace:
I20240501 14:09:04.908210 27609 rpcz_store.cc:276] 0501 14:09:03.898974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 14:09:03.899029 (+ 55us) service_pool.cc:225] Handling call
0501 14:09:03.899727 (+ 698us) log.cc:842] Serialized 263 byte log entry
0501 14:09:04.908071 (+1008344us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":40,"raft.queue_time_us":43,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":153,"spinlock_wait_cycles":1792,"thread_start_us":63,"threads_started":1,"wal-append.queue_time_us":134,"wal-append.run_cpu_time_us":315,"wal-append.run_wall_time_us":1000523}
I20240501 14:09:04.910121 25378 catalog_manager.cc:5484] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 reported cstate change: config changed from index -1 to 4, VOTER 0e3ebe10573a4863b3734daf8f3c4382 (127.22.252.134) added. New cstate: current_term: 1 leader_uuid: "e8f33be94f864561bec2e185b5a695c3" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } } }
I20240501 14:09:04.929845 25660 consensus_queue.cc:237] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } }
W20240501 14:09:04.933979 25596 consensus_peers.cc:485] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d -> Peer 3df2596a705349f185baa3b69dabfc25 (127.22.252.135:39631): Couldn't send request to peer 3df2596a705349f185baa3b69dabfc25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82731ca00eea49248cf51765e441fea6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:05.446684 27636 ts_tablet_manager.cc:902] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25: Initiating tablet copy from peer c6717727de344c0b90dd50fd5e70883d (127.22.252.130:44715)
I20240501 14:09:05.447131 27636 tablet_copy_client.cc:308] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.130:44715
I20240501 14:09:05.450425 25680 tablet_copy_service.cc:140] P c6717727de344c0b90dd50fd5e70883d: Received BeginTabletCopySession request for tablet 82731ca00eea49248cf51765e441fea6 from peer 3df2596a705349f185baa3b69dabfc25 ({username='slave'} at 127.22.252.135:53131)
I20240501 14:09:05.450527 25680 tablet_copy_service.cc:161] P c6717727de344c0b90dd50fd5e70883d: Beginning new tablet copy session on tablet 82731ca00eea49248cf51765e441fea6 from peer 3df2596a705349f185baa3b69dabfc25 at {username='slave'} at 127.22.252.135:53131: session id = 3df2596a705349f185baa3b69dabfc25-82731ca00eea49248cf51765e441fea6
I20240501 14:09:05.451265 25680 tablet_copy_source_session.cc:205] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:05.452180 27636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82731ca00eea49248cf51765e441fea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:05.453938 27636 tablet_copy_client.cc:783] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:05.454085 27636 tablet_copy_client.cc:647] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:05.455088 27636 tablet_copy_client.cc:523] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:05.456010 27636 tablet_bootstrap.cc:492] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25: Bootstrap starting.
I20240501 14:09:05.472600 27636 tablet_bootstrap.cc:492] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:09:05.472847 27636 tablet_bootstrap.cc:492] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25: Bootstrap complete.
I20240501 14:09:05.473003 27636 ts_tablet_manager.cc:1367] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25: Time spent bootstrapping tablet: real 0.017s user 0.017s sys 0.000s
I20240501 14:09:05.473446 27636 raft_consensus.cc:348] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:09:05.473649 27636 raft_consensus.cc:729] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3df2596a705349f185baa3b69dabfc25, State: Initialized, Role: FOLLOWER
I20240501 14:09:05.473773 27636 consensus_queue.cc:260] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } }
I20240501 14:09:05.474112 27636 ts_tablet_manager.cc:1398] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:05.474574 25680 tablet_copy_service.cc:342] P c6717727de344c0b90dd50fd5e70883d: Request end of tablet copy session 3df2596a705349f185baa3b69dabfc25-82731ca00eea49248cf51765e441fea6 received from {username='slave'} at 127.22.252.135:53131
I20240501 14:09:05.474654 25680 tablet_copy_service.cc:433] P c6717727de344c0b90dd50fd5e70883d: ending tablet copy session 3df2596a705349f185baa3b69dabfc25-82731ca00eea49248cf51765e441fea6 on tablet 82731ca00eea49248cf51765e441fea6 with peer 3df2596a705349f185baa3b69dabfc25
I20240501 14:09:05.960788 27194 raft_consensus.cc:1203] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:05.961673 27646 raft_consensus.cc:2932] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3df2596a705349f185baa3b69dabfc25 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } } }
I20240501 14:09:05.962239 27194 raft_consensus.cc:2932] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3df2596a705349f185baa3b69dabfc25 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } } }
I20240501 14:09:05.962963 27646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:59158 (request call id 0) took 1033 ms. Trace:
I20240501 14:09:05.963071 27646 rpcz_store.cc:276] 0501 14:09:04.929526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 14:09:04.929577 (+ 51us) service_pool.cc:225] Handling call
0501 14:09:04.930306 (+ 729us) log.cc:842] Serialized 263 byte log entry
0501 14:09:05.962946 (+1032640us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":37,"raft.queue_time_us":38,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":142,"thread_start_us":62,"threads_started":1,"wal-append.queue_time_us":126,"wal-append.run_cpu_time_us":327,"wal-append.run_wall_time_us":1000386}
I20240501 14:09:05.964339 25378 catalog_manager.cc:5484] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d reported cstate change: config changed from index -1 to 4, VOTER 3df2596a705349f185baa3b69dabfc25 (127.22.252.135) added. New cstate: current_term: 1 leader_uuid: "c6717727de344c0b90dd50fd5e70883d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } } }
I20240501 14:09:06.021298 27041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.022892 26772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.025084 26640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.027141 25509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:06.033742 27174 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.034224 25640 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:06.034237 26912 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.075091 25660 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "82731ca00eea49248cf51765e441fea6"
dest_uuid: "c6717727de344c0b90dd50fd5e70883d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59182
I20240501 14:09:06.075296 25660 raft_consensus.cc:595] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:06.122685 27041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.123836 27174 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.125533 26772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.134099 26912 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.138458 25509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:06.140435 26640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.143476 25640 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:06.220393 27174 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.227687 26640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.227944 25509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:06.230361 26772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.236773 27041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.238204 25640 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:06.249271 26912 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.277472 25529 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78ff5bcb4c5445659d365e84dc810ebc"
dest_uuid: "e8f33be94f864561bec2e185b5a695c3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35978
I20240501 14:09:06.277647 25529 raft_consensus.cc:595] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:06.328956 26772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.329135 27174 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.329623 27041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.336041 25640 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:06.342340 26640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.351540 26912 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.356954 25509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:06.390364 27609 raft_consensus.cc:979] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3: : Instructing follower 0e3ebe10573a4863b3734daf8f3c4382 to start an election
I20240501 14:09:06.390468 27609 raft_consensus.cc:1067] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Signalling peer 0e3ebe10573a4863b3734daf8f3c4382 to start an election
I20240501 14:09:06.390998 27063 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78ff5bcb4c5445659d365e84dc810ebc"
dest_uuid: "0e3ebe10573a4863b3734daf8f3c4382"
from {username='slave'} at 127.22.252.129:54277
I20240501 14:09:06.391183 27063 raft_consensus.cc:482] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:06.391304 27063 raft_consensus.cc:3037] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:06.392107 27063 raft_consensus.cc:504] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } }
I20240501 14:09:06.392971 27063 leader_election.cc:288] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382 [CANDIDATE]: Term 2 election: Requested vote from peers e8f33be94f864561bec2e185b5a695c3 (127.22.252.129:44353)
I20240501 14:09:06.393349 25529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff5bcb4c5445659d365e84dc810ebc" candidate_uuid: "0e3ebe10573a4863b3734daf8f3c4382" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e8f33be94f864561bec2e185b5a695c3"
I20240501 14:09:06.393522 25529 raft_consensus.cc:3032] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:06.393563 25529 raft_consensus.cc:729] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8f33be94f864561bec2e185b5a695c3, State: Running, Role: LEADER
I20240501 14:09:06.393695 25529 consensus_queue.cc:260] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } }
I20240501 14:09:06.393895 25529 raft_consensus.cc:3037] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:06.394719 25529 raft_consensus.cc:2454] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e3ebe10573a4863b3734daf8f3c4382 in term 2.
I20240501 14:09:06.395092 26999 leader_election.cc:302] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e3ebe10573a4863b3734daf8f3c4382, e8f33be94f864561bec2e185b5a695c3; no voters:
I20240501 14:09:06.395426 27821 raft_consensus.cc:2781] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:06.395591 27821 raft_consensus.cc:686] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382 [term 2 LEADER]: Becoming Leader. State: Replica: 0e3ebe10573a4863b3734daf8f3c4382, State: Running, Role: LEADER
I20240501 14:09:06.395890 27821 consensus_queue.cc:237] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } }
I20240501 14:09:06.398079 25387 catalog_manager.cc:5484] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382 reported cstate change: term changed from 1 to 2, leader changed from e8f33be94f864561bec2e185b5a695c3 (127.22.252.129) to 0e3ebe10573a4863b3734daf8f3c4382 (127.22.252.134). New cstate: current_term: 2 leader_uuid: "0e3ebe10573a4863b3734daf8f3c4382" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } } }
I20240501 14:09:06.436268 27041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.437772 26640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.442551 25509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:06.444411 27174 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.455545 26912 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.458544 25640 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:06.459030 26772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.490465 25529 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d16f3d9f72b44c2cb15a6c634bee2a28"
dest_uuid: "e8f33be94f864561bec2e185b5a695c3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36016
I20240501 14:09:06.490592 25529 raft_consensus.cc:595] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:06.521617 27646 raft_consensus.cc:979] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d: : Instructing follower 3df2596a705349f185baa3b69dabfc25 to start an election
I20240501 14:09:06.521721 27646 raft_consensus.cc:1067] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Signalling peer 3df2596a705349f185baa3b69dabfc25 to start an election
I20240501 14:09:06.522333 27194 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "82731ca00eea49248cf51765e441fea6"
dest_uuid: "3df2596a705349f185baa3b69dabfc25"
from {username='slave'} at 127.22.252.130:42713
I20240501 14:09:06.522503 27194 raft_consensus.cc:482] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:06.522600 27194 raft_consensus.cc:3037] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:06.523459 27194 raft_consensus.cc:504] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } }
I20240501 14:09:06.524381 27194 leader_election.cc:288] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25 [CANDIDATE]: Term 2 election: Requested vote from peers c6717727de344c0b90dd50fd5e70883d (127.22.252.130:44715)
I20240501 14:09:06.524763 25660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82731ca00eea49248cf51765e441fea6" candidate_uuid: "3df2596a705349f185baa3b69dabfc25" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c6717727de344c0b90dd50fd5e70883d"
I20240501 14:09:06.524899 25660 raft_consensus.cc:3032] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:06.524950 25660 raft_consensus.cc:729] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c6717727de344c0b90dd50fd5e70883d, State: Running, Role: LEADER
I20240501 14:09:06.525135 25660 consensus_queue.cc:260] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } }
I20240501 14:09:06.525355 25660 raft_consensus.cc:3037] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:06.526181 25660 raft_consensus.cc:2454] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3df2596a705349f185baa3b69dabfc25 in term 2.
I20240501 14:09:06.526540 27129 leader_election.cc:302] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3df2596a705349f185baa3b69dabfc25, c6717727de344c0b90dd50fd5e70883d; no voters:
I20240501 14:09:06.526881 27879 raft_consensus.cc:2781] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:06.527053 27879 raft_consensus.cc:686] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25 [term 2 LEADER]: Becoming Leader. State: Replica: 3df2596a705349f185baa3b69dabfc25, State: Running, Role: LEADER
I20240501 14:09:06.527400 27879 consensus_queue.cc:237] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } }
I20240501 14:09:06.531023 25378 catalog_manager.cc:5484] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25 reported cstate change: term changed from 1 to 2, leader changed from c6717727de344c0b90dd50fd5e70883d (127.22.252.130) to 3df2596a705349f185baa3b69dabfc25 (127.22.252.135). New cstate: current_term: 2 leader_uuid: "3df2596a705349f185baa3b69dabfc25" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } } }
I20240501 14:09:06.541530 27041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.544885 27174 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.545178 26772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.550812 25509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:06.553120 25640 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:06.558358 26640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.573038 26912 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.643613 27174 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.645725 26640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.648428 25640 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:06.650797 27041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.663192 26912 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.664973 25509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:06.678933 26772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.698297 25660 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fd9a83b1cc1441da972fb41466aaf402"
dest_uuid: "c6717727de344c0b90dd50fd5e70883d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59264
I20240501 14:09:06.698434 25660 raft_consensus.cc:595] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:06.744347 27041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.749091 25640 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:06.750510 26772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.753401 25509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:06.762269 26640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.762578 26912 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.762792 27174 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.850807 26772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.851523 27041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.856590 26912 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.862470 26640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.863370 25509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:06.864521 27174 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.866812 25640 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:06.875599 27820 raft_consensus.cc:979] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3: : Instructing follower 8a9c183caefa4437b53752c1203b8bb7 to start an election
I20240501 14:09:06.875695 27820 raft_consensus.cc:1067] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Signalling peer 8a9c183caefa4437b53752c1203b8bb7 to start an election
I20240501 14:09:06.876168 26660 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d16f3d9f72b44c2cb15a6c634bee2a28"
dest_uuid: "8a9c183caefa4437b53752c1203b8bb7"
from {username='slave'} at 127.22.252.129:35227
I20240501 14:09:06.876367 26660 raft_consensus.cc:482] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:06.876458 26660 raft_consensus.cc:3037] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:06.877485 26660 raft_consensus.cc:504] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } }
I20240501 14:09:06.878487 26660 leader_election.cc:288] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7 [CANDIDATE]: Term 2 election: Requested vote from peers e8f33be94f864561bec2e185b5a695c3 (127.22.252.129:44353)
I20240501 14:09:06.879024 25529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16f3d9f72b44c2cb15a6c634bee2a28" candidate_uuid: "8a9c183caefa4437b53752c1203b8bb7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e8f33be94f864561bec2e185b5a695c3"
I20240501 14:09:06.879133 25529 raft_consensus.cc:3032] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:06.879173 25529 raft_consensus.cc:729] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8f33be94f864561bec2e185b5a695c3, State: Running, Role: LEADER
I20240501 14:09:06.879295 25529 consensus_queue.cc:260] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } }
I20240501 14:09:06.879384 25529 raft_consensus.cc:3037] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:06.880285 25529 raft_consensus.cc:2454] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c183caefa4437b53752c1203b8bb7 in term 2.
I20240501 14:09:06.880640 26596 leader_election.cc:302] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8a9c183caefa4437b53752c1203b8bb7, e8f33be94f864561bec2e185b5a695c3; no voters:
I20240501 14:09:06.881532 25529 raft_consensus.cc:1261] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Refusing update from remote peer 0e3ebe10573a4863b3734daf8f3c4382: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:06.882126 27821 consensus_queue.cc:1035] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:06.882889 28002 raft_consensus.cc:2781] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:06.883124 28002 raft_consensus.cc:686] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7 [term 2 LEADER]: Becoming Leader. State: Replica: 8a9c183caefa4437b53752c1203b8bb7, State: Running, Role: LEADER
I20240501 14:09:06.883494 28002 consensus_queue.cc:237] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } }
I20240501 14:09:06.886390 25379 catalog_manager.cc:5484] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7 reported cstate change: term changed from 1 to 2, leader changed from e8f33be94f864561bec2e185b5a695c3 (127.22.252.129) to 8a9c183caefa4437b53752c1203b8bb7 (127.22.252.131). New cstate: current_term: 2 leader_uuid: "8a9c183caefa4437b53752c1203b8bb7" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } } }
I20240501 14:09:06.902086 25529 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "36b4476375e546298f3d8184dda64204"
dest_uuid: "e8f33be94f864561bec2e185b5a695c3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36060
I20240501 14:09:06.902197 25529 raft_consensus.cc:595] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:06.950403 26772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.956616 25509 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:06.957509 25640 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:06.958067 27041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.959565 26912 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:06.964421 26640 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.976691 27174 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:06.977700 27646 raft_consensus.cc:979] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d: : Instructing follower e9e15d9915264a62a5c9e68f67872199 to start an election
I20240501 14:09:06.977774 27646 raft_consensus.cc:1067] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Signalling peer e9e15d9915264a62a5c9e68f67872199 to start an election
I20240501 14:09:06.978601 25660 raft_consensus.cc:1261] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [term 2 FOLLOWER]: Refusing update from remote peer 3df2596a705349f185baa3b69dabfc25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:06.979120 26932 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fd9a83b1cc1441da972fb41466aaf402"
dest_uuid: "e9e15d9915264a62a5c9e68f67872199"
from {username='slave'} at 127.22.252.130:39273
I20240501 14:09:06.979285 26932 raft_consensus.cc:482] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:06.979391 26932 raft_consensus.cc:3037] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:06.980540 26932 raft_consensus.cc:504] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } }
I20240501 14:09:06.981462 26932 leader_election.cc:288] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199 [CANDIDATE]: Term 2 election: Requested vote from peers c6717727de344c0b90dd50fd5e70883d (127.22.252.130:44715)
I20240501 14:09:06.982189 25660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd9a83b1cc1441da972fb41466aaf402" candidate_uuid: "e9e15d9915264a62a5c9e68f67872199" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c6717727de344c0b90dd50fd5e70883d"
I20240501 14:09:06.982304 25660 raft_consensus.cc:3032] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:06.982340 25660 raft_consensus.cc:729] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c6717727de344c0b90dd50fd5e70883d, State: Running, Role: LEADER
I20240501 14:09:06.982281 27879 consensus_queue.cc:1035] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:06.982465 25660 consensus_queue.cc:260] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } }
I20240501 14:09:06.982592 25660 raft_consensus.cc:3037] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:06.983572 25660 raft_consensus.cc:2454] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9e15d9915264a62a5c9e68f67872199 in term 2.
I20240501 14:09:06.984316 26867 leader_election.cc:302] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c6717727de344c0b90dd50fd5e70883d, e9e15d9915264a62a5c9e68f67872199; no voters:
I20240501 14:09:06.984817 28046 raft_consensus.cc:2781] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:06.985026 28046 raft_consensus.cc:686] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199 [term 2 LEADER]: Becoming Leader. State: Replica: e9e15d9915264a62a5c9e68f67872199, State: Running, Role: LEADER
I20240501 14:09:06.985536 28046 consensus_queue.cc:237] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } }
I20240501 14:09:06.996268 25383 catalog_manager.cc:5484] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199 reported cstate change: term changed from 1 to 2, leader changed from c6717727de344c0b90dd50fd5e70883d (127.22.252.130) to e9e15d9915264a62a5c9e68f67872199 (127.22.252.133). New cstate: current_term: 2 leader_uuid: "e9e15d9915264a62a5c9e68f67872199" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } } }
I20240501 14:09:07.069227 27041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.069975 27174 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.073097 26772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.073997 25509 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:07.077481 25640 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:07.079556 26640 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.103569 26912 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.124678 25659 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a58883c2a8074e01b2170310e3008e52"
dest_uuid: "c6717727de344c0b90dd50fd5e70883d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59326
I20240501 14:09:07.124852 25659 raft_consensus.cc:595] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:07.178995 27041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.180042 27174 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.180413 26640 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.181281 25640 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:07.188995 25509 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:07.189863 26772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.215919 26912 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.286491 26772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.287079 27174 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.292256 27041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.292869 25640 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:07.293622 26640 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.312047 26912 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.317565 27609 raft_consensus.cc:979] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3: : Instructing follower 3df2596a705349f185baa3b69dabfc25 to start an election
I20240501 14:09:07.317646 27609 raft_consensus.cc:1067] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Signalling peer 3df2596a705349f185baa3b69dabfc25 to start an election
I20240501 14:09:07.318156 27194 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "36b4476375e546298f3d8184dda64204"
dest_uuid: "3df2596a705349f185baa3b69dabfc25"
from {username='slave'} at 127.22.252.129:38909
I20240501 14:09:07.318313 27194 raft_consensus.cc:482] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:07.318363 27194 raft_consensus.cc:3037] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:07.319367 27194 raft_consensus.cc:504] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } }
I20240501 14:09:07.319733 27194 leader_election.cc:288] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25 [CANDIDATE]: Term 2 election: Requested vote from peers e8f33be94f864561bec2e185b5a695c3 (127.22.252.129:44353)
I20240501 14:09:07.320096 25529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36b4476375e546298f3d8184dda64204" candidate_uuid: "3df2596a705349f185baa3b69dabfc25" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e8f33be94f864561bec2e185b5a695c3"
I20240501 14:09:07.320262 25529 raft_consensus.cc:3032] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:07.320317 25529 raft_consensus.cc:729] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8f33be94f864561bec2e185b5a695c3, State: Running, Role: LEADER
I20240501 14:09:07.320436 25529 consensus_queue.cc:260] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } }
I20240501 14:09:07.320549 25529 raft_consensus.cc:3037] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:07.321470 25529 raft_consensus.cc:2454] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3df2596a705349f185baa3b69dabfc25 in term 2.
I20240501 14:09:07.322377 27130 leader_election.cc:302] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3df2596a705349f185baa3b69dabfc25, e8f33be94f864561bec2e185b5a695c3; no voters:
I20240501 14:09:07.322563 27880 raft_consensus.cc:2781] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:07.322646 27880 raft_consensus.cc:686] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25 [term 2 LEADER]: Becoming Leader. State: Replica: 3df2596a705349f185baa3b69dabfc25, State: Running, Role: LEADER
I20240501 14:09:07.322773 27880 consensus_queue.cc:237] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } }
I20240501 14:09:07.324586 25380 catalog_manager.cc:5484] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25 reported cstate change: term changed from 1 to 2, leader changed from e8f33be94f864561bec2e185b5a695c3 (127.22.252.129) to 3df2596a705349f185baa3b69dabfc25 (127.22.252.135). New cstate: current_term: 2 leader_uuid: "3df2596a705349f185baa3b69dabfc25" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } } }
I20240501 14:09:07.332600 25509 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:07.352571 25529 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8a1f7cc1afb143a8a132944682f969fe"
dest_uuid: "e8f33be94f864561bec2e185b5a695c3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36114
I20240501 14:09:07.352713 25529 raft_consensus.cc:595] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:07.404732 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:07.404886 27041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.406042 26772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.407097 25640 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:07.413645 26640 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.414103 25509 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:07.434154 26912 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.452651 25529 raft_consensus.cc:1261] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Refusing update from remote peer 8a9c183caefa4437b53752c1203b8bb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:07.453168 28202 consensus_queue.cc:1035] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:07.509661 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:07.510871 26772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.511794 27041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.516808 25640 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:07.522130 26912 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.525246 26640 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.532876 25509 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:07.557260 25659 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e9029b14eda48cab26e246ed7535d65"
dest_uuid: "c6717727de344c0b90dd50fd5e70883d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59396
I20240501 14:09:07.557423 25659 raft_consensus.cc:595] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:07.566509 25659 raft_consensus.cc:1261] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [term 2 FOLLOWER]: Refusing update from remote peer e9e15d9915264a62a5c9e68f67872199: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:07.567009 28255 consensus_queue.cc:1035] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:07.619544 27041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.622378 25509 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:07.622962 25640 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:07.623385 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:07.628854 27878 raft_consensus.cc:979] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d: : Instructing follower 0e3ebe10573a4863b3734daf8f3c4382 to start an election
I20240501 14:09:07.628863 26772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.628955 27878 raft_consensus.cc:1067] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Signalling peer 0e3ebe10573a4863b3734daf8f3c4382 to start an election
I20240501 14:09:07.629552 27063 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a58883c2a8074e01b2170310e3008e52"
dest_uuid: "0e3ebe10573a4863b3734daf8f3c4382"
from {username='slave'} at 127.22.252.130:50975
I20240501 14:09:07.629683 27063 raft_consensus.cc:482] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:07.629734 27063 raft_consensus.cc:3037] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:07.630754 27063 raft_consensus.cc:504] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } }
I20240501 14:09:07.631130 27063 leader_election.cc:288] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382 [CANDIDATE]: Term 2 election: Requested vote from peers c6717727de344c0b90dd50fd5e70883d (127.22.252.130:44715)
I20240501 14:09:07.631511 25659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58883c2a8074e01b2170310e3008e52" candidate_uuid: "0e3ebe10573a4863b3734daf8f3c4382" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c6717727de344c0b90dd50fd5e70883d"
I20240501 14:09:07.631630 25659 raft_consensus.cc:3032] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:07.631666 25659 raft_consensus.cc:729] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c6717727de344c0b90dd50fd5e70883d, State: Running, Role: LEADER
I20240501 14:09:07.631803 25659 consensus_queue.cc:260] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } }
I20240501 14:09:07.631937 25659 raft_consensus.cc:3037] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:07.633419 25659 raft_consensus.cc:2454] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e3ebe10573a4863b3734daf8f3c4382 in term 2.
I20240501 14:09:07.633728 26998 leader_election.cc:302] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e3ebe10573a4863b3734daf8f3c4382, c6717727de344c0b90dd50fd5e70883d; no voters:
I20240501 14:09:07.633957 28201 raft_consensus.cc:2781] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:07.634048 28201 raft_consensus.cc:686] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382 [term 2 LEADER]: Becoming Leader. State: Replica: 0e3ebe10573a4863b3734daf8f3c4382, State: Running, Role: LEADER
I20240501 14:09:07.634171 28201 consensus_queue.cc:237] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } }
I20240501 14:09:07.634399 26640 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.636150 25378 catalog_manager.cc:5484] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382 reported cstate change: term changed from 1 to 2, leader changed from c6717727de344c0b90dd50fd5e70883d (127.22.252.130) to 0e3ebe10573a4863b3734daf8f3c4382 (127.22.252.134). New cstate: current_term: 2 leader_uuid: "0e3ebe10573a4863b3734daf8f3c4382" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } } }
I20240501 14:09:07.656055 26912 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.704687 25529 raft_consensus.cc:1261] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Refusing update from remote peer 3df2596a705349f185baa3b69dabfc25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:07.705174 27879 consensus_queue.cc:1035] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:07.733893 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:07.735949 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:07.737946 26640 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.739774 25640 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:07.747205 28287 raft_consensus.cc:979] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d: : Instructing follower 8a9c183caefa4437b53752c1203b8bb7 to start an election
I20240501 14:09:07.747293 28287 raft_consensus.cc:1067] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Signalling peer 8a9c183caefa4437b53752c1203b8bb7 to start an election
I20240501 14:09:07.747774 26660 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e9029b14eda48cab26e246ed7535d65"
dest_uuid: "8a9c183caefa4437b53752c1203b8bb7"
from {username='slave'} at 127.22.252.130:36409
I20240501 14:09:07.747902 26660 raft_consensus.cc:482] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:07.747956 26660 raft_consensus.cc:3037] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:07.749063 26660 raft_consensus.cc:504] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } }
I20240501 14:09:07.749418 26660 leader_election.cc:288] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7 [CANDIDATE]: Term 2 election: Requested vote from peers c6717727de344c0b90dd50fd5e70883d (127.22.252.130:44715)
I20240501 14:09:07.751636 25659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e9029b14eda48cab26e246ed7535d65" candidate_uuid: "8a9c183caefa4437b53752c1203b8bb7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c6717727de344c0b90dd50fd5e70883d"
I20240501 14:09:07.751765 25659 raft_consensus.cc:3032] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:07.751811 25659 raft_consensus.cc:729] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c6717727de344c0b90dd50fd5e70883d, State: Running, Role: LEADER
I20240501 14:09:07.751942 25659 consensus_queue.cc:260] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } }
I20240501 14:09:07.752084 25659 raft_consensus.cc:3037] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:07.753103 25659 raft_consensus.cc:2454] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9c183caefa4437b53752c1203b8bb7 in term 2.
I20240501 14:09:07.753428 26595 leader_election.cc:302] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8a9c183caefa4437b53752c1203b8bb7, c6717727de344c0b90dd50fd5e70883d; no voters:
I20240501 14:09:07.753654 28209 raft_consensus.cc:2781] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:07.753718 28209 raft_consensus.cc:686] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7 [term 2 LEADER]: Becoming Leader. State: Replica: 8a9c183caefa4437b53752c1203b8bb7, State: Running, Role: LEADER
I20240501 14:09:07.753825 28209 consensus_queue.cc:237] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } }
I20240501 14:09:07.755628 25386 catalog_manager.cc:5484] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7 reported cstate change: term changed from 1 to 2, leader changed from c6717727de344c0b90dd50fd5e70883d (127.22.252.130) to 8a9c183caefa4437b53752c1203b8bb7 (127.22.252.131). New cstate: current_term: 2 leader_uuid: "8a9c183caefa4437b53752c1203b8bb7" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } } }
I20240501 14:09:07.756992 26912 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.766377 26772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.766430 25509 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:07.792179 25529 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c230502585ed4dd1aa75bc02a3e7d242"
dest_uuid: "e8f33be94f864561bec2e185b5a695c3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51504
I20240501 14:09:07.792366 25529 raft_consensus.cc:595] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:07.847152 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:07.848263 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:07.854799 25509 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:07.860549 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:07.862728 25640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:07.866945 26772 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.882934 26912 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.928984 27820 raft_consensus.cc:979] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3: : Instructing follower 86b991e2c8414b6698e51851d34bc8f2 to start an election
I20240501 14:09:07.929141 27820 raft_consensus.cc:1067] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Signalling peer 86b991e2c8414b6698e51851d34bc8f2 to start an election
I20240501 14:09:07.929656 26792 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8a1f7cc1afb143a8a132944682f969fe"
dest_uuid: "86b991e2c8414b6698e51851d34bc8f2"
from {username='slave'} at 127.22.252.129:44969
I20240501 14:09:07.929867 26792 raft_consensus.cc:482] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:07.929939 26792 raft_consensus.cc:3037] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:07.930796 26792 raft_consensus.cc:504] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "86b991e2c8414b6698e51851d34bc8f2" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 37205 } }
I20240501 14:09:07.931532 26792 leader_election.cc:288] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2 [CANDIDATE]: Term 2 election: Requested vote from peers e8f33be94f864561bec2e185b5a695c3 (127.22.252.129:44353)
I20240501 14:09:07.931948 25529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a1f7cc1afb143a8a132944682f969fe" candidate_uuid: "86b991e2c8414b6698e51851d34bc8f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e8f33be94f864561bec2e185b5a695c3"
I20240501 14:09:07.932071 25529 raft_consensus.cc:3032] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:07.932119 25529 raft_consensus.cc:729] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8f33be94f864561bec2e185b5a695c3, State: Running, Role: LEADER
I20240501 14:09:07.932308 25529 consensus_queue.cc:260] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "86b991e2c8414b6698e51851d34bc8f2" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 37205 } }
I20240501 14:09:07.932448 25529 raft_consensus.cc:3037] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:07.933313 25529 raft_consensus.cc:2454] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86b991e2c8414b6698e51851d34bc8f2 in term 2.
I20240501 14:09:07.933676 26727 leader_election.cc:302] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 86b991e2c8414b6698e51851d34bc8f2, e8f33be94f864561bec2e185b5a695c3; no voters:
I20240501 14:09:07.933962 28367 raft_consensus.cc:2781] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:07.934123 28367 raft_consensus.cc:686] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2 [term 2 LEADER]: Becoming Leader. State: Replica: 86b991e2c8414b6698e51851d34bc8f2, State: Running, Role: LEADER
I20240501 14:09:07.934264 28367 consensus_queue.cc:237] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "86b991e2c8414b6698e51851d34bc8f2" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 37205 } }
I20240501 14:09:07.935972 25383 catalog_manager.cc:5484] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2 reported cstate change: term changed from 1 to 2, leader changed from e8f33be94f864561bec2e185b5a695c3 (127.22.252.129) to 86b991e2c8414b6698e51851d34bc8f2 (127.22.252.132). New cstate: current_term: 2 leader_uuid: "86b991e2c8414b6698e51851d34bc8f2" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "86b991e2c8414b6698e51851d34bc8f2" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 37205 } } }
I20240501 14:09:07.951711 28366 raft_consensus.cc:979] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3: : Instructing follower e9e15d9915264a62a5c9e68f67872199 to start an election
I20240501 14:09:07.951853 28366 raft_consensus.cc:1067] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Signalling peer e9e15d9915264a62a5c9e68f67872199 to start an election
I20240501 14:09:07.952303 26932 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c230502585ed4dd1aa75bc02a3e7d242"
dest_uuid: "e9e15d9915264a62a5c9e68f67872199"
from {username='slave'} at 127.22.252.129:49801
I20240501 14:09:07.952445 26932 raft_consensus.cc:482] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:07.952512 26932 raft_consensus.cc:3037] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:07.953369 26932 raft_consensus.cc:504] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } }
I20240501 14:09:07.953711 26932 leader_election.cc:288] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199 [CANDIDATE]: Term 2 election: Requested vote from peers e8f33be94f864561bec2e185b5a695c3 (127.22.252.129:44353)
I20240501 14:09:07.954205 25529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c230502585ed4dd1aa75bc02a3e7d242" candidate_uuid: "e9e15d9915264a62a5c9e68f67872199" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e8f33be94f864561bec2e185b5a695c3"
I20240501 14:09:07.954322 25529 raft_consensus.cc:3032] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:07.954375 25529 raft_consensus.cc:729] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8f33be94f864561bec2e185b5a695c3, State: Running, Role: LEADER
I20240501 14:09:07.954516 25529 consensus_queue.cc:260] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } }
I20240501 14:09:07.954644 25529 raft_consensus.cc:3037] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:07.955402 25529 raft_consensus.cc:2454] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9e15d9915264a62a5c9e68f67872199 in term 2.
I20240501 14:09:07.955688 26868 leader_election.cc:302] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: e8f33be94f864561bec2e185b5a695c3, e9e15d9915264a62a5c9e68f67872199; no voters:
I20240501 14:09:07.955858 28257 raft_consensus.cc:2781] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:07.955945 28257 raft_consensus.cc:686] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199 [term 2 LEADER]: Becoming Leader. State: Replica: e9e15d9915264a62a5c9e68f67872199, State: Running, Role: LEADER
I20240501 14:09:07.956079 28257 consensus_queue.cc:237] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } }
I20240501 14:09:07.957546 25383 catalog_manager.cc:5484] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199 reported cstate change: term changed from 1 to 2, leader changed from e8f33be94f864561bec2e185b5a695c3 (127.22.252.129) to e9e15d9915264a62a5c9e68f67872199 (127.22.252.133). New cstate: current_term: 2 leader_uuid: "e9e15d9915264a62a5c9e68f67872199" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } } }
I20240501 14:09:08.159191 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.163493 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.166399 25640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:08.167222 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.167882 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.174741 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.182212 25509 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:08.224540 27194 consensus_queue.cc:237] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } }
I20240501 14:09:08.225224 27880 raft_consensus.cc:2932] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER c6717727de344c0b90dd50fd5e70883d (127.22.252.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } } }
I20240501 14:09:08.227921 25380 catalog_manager.cc:5484] T 82731ca00eea49248cf51765e441fea6 P 3df2596a705349f185baa3b69dabfc25 reported cstate change: config changed from index 4 to 6, VOTER c6717727de344c0b90dd50fd5e70883d (127.22.252.130) evicted. New cstate: current_term: 2 leader_uuid: "3df2596a705349f185baa3b69dabfc25" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } } }
I20240501 14:09:08.230782 25640 tablet_service.cc:1514] Processing DeleteTablet for tablet 82731ca00eea49248cf51765e441fea6 with delete_type TABLET_DATA_TOMBSTONED (TS c6717727de344c0b90dd50fd5e70883d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60440
I20240501 14:09:08.231256 28430 tablet_replica.cc:330] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d: stopping tablet replica
I20240501 14:09:08.231519 28430 raft_consensus.cc:2229] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:08.231664 28430 raft_consensus.cc:2258] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:08.232638 28430 ts_tablet_manager.cc:1875] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:08.234687 28430 ts_tablet_manager.cc:1888] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 14:09:08.234786 28430 log.cc:1196] T 82731ca00eea49248cf51765e441fea6 P c6717727de344c0b90dd50fd5e70883d: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/82731ca00eea49248cf51765e441fea6
I20240501 14:09:08.235764 25373 catalog_manager.cc:4830] TS c6717727de344c0b90dd50fd5e70883d (127.22.252.130:44715): tablet 82731ca00eea49248cf51765e441fea6 (table rebalance_test_table_3 [id=af481f6478684cf7ab01dc10b39463a7]) successfully deleted
I20240501 14:09:08.236786 25659 raft_consensus.cc:1261] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [term 2 FOLLOWER]: Refusing update from remote peer 0e3ebe10573a4863b3734daf8f3c4382: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:08.237177 28200 consensus_queue.cc:1035] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:08.283051 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.283566 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.284847 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.287829 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.289664 25640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:08.290637 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.306264 25509 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:08.349491 27063 consensus_queue.cc:237] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } }
I20240501 14:09:08.350239 28438 raft_consensus.cc:2932] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER e8f33be94f864561bec2e185b5a695c3 (127.22.252.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } } }
I20240501 14:09:08.352527 25379 catalog_manager.cc:5484] T 78ff5bcb4c5445659d365e84dc810ebc P 0e3ebe10573a4863b3734daf8f3c4382 reported cstate change: config changed from index 4 to 6, VOTER e8f33be94f864561bec2e185b5a695c3 (127.22.252.129) evicted. New cstate: current_term: 2 leader_uuid: "0e3ebe10573a4863b3734daf8f3c4382" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } } }
I20240501 14:09:08.355173 25509 tablet_service.cc:1514] Processing DeleteTablet for tablet 78ff5bcb4c5445659d365e84dc810ebc with delete_type TABLET_DATA_TOMBSTONED (TS e8f33be94f864561bec2e185b5a695c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44734
I20240501 14:09:08.355674 28488 tablet_replica.cc:330] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3: stopping tablet replica
I20240501 14:09:08.355994 28488 raft_consensus.cc:2229] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:08.356161 28488 raft_consensus.cc:2258] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:08.357242 28488 ts_tablet_manager.cc:1875] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:08.359282 28488 ts_tablet_manager.cc:1888] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 14:09:08.359359 28488 log.cc:1196] T 78ff5bcb4c5445659d365e84dc810ebc P e8f33be94f864561bec2e185b5a695c3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/78ff5bcb4c5445659d365e84dc810ebc
I20240501 14:09:08.360072 25374 catalog_manager.cc:4830] TS e8f33be94f864561bec2e185b5a695c3 (127.22.252.129:44353): tablet 78ff5bcb4c5445659d365e84dc810ebc (table rebalance_test_table_4 [id=0622125ee1f04076a30c4aa37417bf64]) successfully deleted
I20240501 14:09:08.367267 25659 raft_consensus.cc:1261] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [term 2 FOLLOWER]: Refusing update from remote peer 8a9c183caefa4437b53752c1203b8bb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:08.367642 28202 consensus_queue.cc:1035] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:08.406513 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.406888 25640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:08.407680 25529 raft_consensus.cc:1261] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Refusing update from remote peer e9e15d9915264a62a5c9e68f67872199: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:08.408257 28257 consensus_queue.cc:1035] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:08.410054 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.419756 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.419790 25528 raft_consensus.cc:1261] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Refusing update from remote peer 86b991e2c8414b6698e51851d34bc8f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:08.424052 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.424556 28367 consensus_queue.cc:1035] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:08.425372 25509 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:08.442157 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.477347 26660 consensus_queue.cc:237] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } }
I20240501 14:09:08.478067 28496 raft_consensus.cc:2932] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER e8f33be94f864561bec2e185b5a695c3 (127.22.252.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } } }
I20240501 14:09:08.480675 25387 catalog_manager.cc:5484] T d16f3d9f72b44c2cb15a6c634bee2a28 P 8a9c183caefa4437b53752c1203b8bb7 reported cstate change: config changed from index 4 to 6, VOTER e8f33be94f864561bec2e185b5a695c3 (127.22.252.129) evicted. New cstate: current_term: 2 leader_uuid: "8a9c183caefa4437b53752c1203b8bb7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } } }
I20240501 14:09:08.482942 25509 tablet_service.cc:1514] Processing DeleteTablet for tablet d16f3d9f72b44c2cb15a6c634bee2a28 with delete_type TABLET_DATA_TOMBSTONED (TS e8f33be94f864561bec2e185b5a695c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44734
I20240501 14:09:08.483297 28488 tablet_replica.cc:330] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3: stopping tablet replica
I20240501 14:09:08.483431 28488 raft_consensus.cc:2229] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:08.483530 28488 raft_consensus.cc:2258] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:08.484040 28488 ts_tablet_manager.cc:1875] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:08.486056 28488 ts_tablet_manager.cc:1888] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 14:09:08.486130 28488 log.cc:1196] T d16f3d9f72b44c2cb15a6c634bee2a28 P e8f33be94f864561bec2e185b5a695c3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/d16f3d9f72b44c2cb15a6c634bee2a28
I20240501 14:09:08.486614 25374 catalog_manager.cc:4830] TS e8f33be94f864561bec2e185b5a695c3 (127.22.252.129:44353): tablet d16f3d9f72b44c2cb15a6c634bee2a28 (table rebalance_test_table_4 [id=0622125ee1f04076a30c4aa37417bf64]) successfully deleted
I20240501 14:09:08.537370 25640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:08.538998 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.541859 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.545830 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.551975 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.552925 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.553378 25509 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:08.601193 26932 consensus_queue.cc:237] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } }
I20240501 14:09:08.601919 28255 raft_consensus.cc:2932] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER c6717727de344c0b90dd50fd5e70883d (127.22.252.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } } }
I20240501 14:09:08.604529 25387 catalog_manager.cc:5484] T fd9a83b1cc1441da972fb41466aaf402 P e9e15d9915264a62a5c9e68f67872199 reported cstate change: config changed from index 4 to 6, VOTER c6717727de344c0b90dd50fd5e70883d (127.22.252.130) evicted. New cstate: current_term: 2 leader_uuid: "e9e15d9915264a62a5c9e68f67872199" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } } }
I20240501 14:09:08.607288 25640 tablet_service.cc:1514] Processing DeleteTablet for tablet fd9a83b1cc1441da972fb41466aaf402 with delete_type TABLET_DATA_TOMBSTONED (TS c6717727de344c0b90dd50fd5e70883d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60440
I20240501 14:09:08.607507 28430 tablet_replica.cc:330] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d: stopping tablet replica
I20240501 14:09:08.607633 28430 raft_consensus.cc:2229] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:08.607738 28430 raft_consensus.cc:2258] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:08.608335 28430 ts_tablet_manager.cc:1875] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:08.610287 28430 ts_tablet_manager.cc:1888] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 14:09:08.610348 28430 log.cc:1196] T fd9a83b1cc1441da972fb41466aaf402 P c6717727de344c0b90dd50fd5e70883d: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/fd9a83b1cc1441da972fb41466aaf402
I20240501 14:09:08.610862 25373 catalog_manager.cc:4830] TS c6717727de344c0b90dd50fd5e70883d (127.22.252.130:44715): tablet fd9a83b1cc1441da972fb41466aaf402 (table rebalance_test_table_1 [id=7f86e635baf845c191b673d7a95b523a]) successfully deleted
I20240501 14:09:08.657822 25640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:08.660797 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.668393 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.668411 25509 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:08.669376 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.669735 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.676980 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.722673 27194 consensus_queue.cc:237] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } }
I20240501 14:09:08.723361 27880 raft_consensus.cc:2932] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER e8f33be94f864561bec2e185b5a695c3 (127.22.252.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } } }
I20240501 14:09:08.725756 25383 catalog_manager.cc:5484] T 36b4476375e546298f3d8184dda64204 P 3df2596a705349f185baa3b69dabfc25 reported cstate change: config changed from index 4 to 6, VOTER e8f33be94f864561bec2e185b5a695c3 (127.22.252.129) evicted. New cstate: current_term: 2 leader_uuid: "3df2596a705349f185baa3b69dabfc25" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3df2596a705349f185baa3b69dabfc25" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 39631 } } }
I20240501 14:09:08.728487 25509 tablet_service.cc:1514] Processing DeleteTablet for tablet 36b4476375e546298f3d8184dda64204 with delete_type TABLET_DATA_TOMBSTONED (TS e8f33be94f864561bec2e185b5a695c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44734
I20240501 14:09:08.728704 28488 tablet_replica.cc:330] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3: stopping tablet replica
I20240501 14:09:08.728870 28488 raft_consensus.cc:2229] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:08.728999 28488 raft_consensus.cc:2258] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:08.729496 28488 ts_tablet_manager.cc:1875] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:08.731616 28488 ts_tablet_manager.cc:1888] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 14:09:08.731717 28488 log.cc:1196] T 36b4476375e546298f3d8184dda64204 P e8f33be94f864561bec2e185b5a695c3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/36b4476375e546298f3d8184dda64204
I20240501 14:09:08.732280 25374 catalog_manager.cc:4830] TS e8f33be94f864561bec2e185b5a695c3 (127.22.252.129:44353): tablet 36b4476375e546298f3d8184dda64204 (table rebalance_test_table_0 [id=aca60ec23c5145babbcbeaaa484c4348]) successfully deleted
I20240501 14:09:08.776510 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.778463 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.781620 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.786492 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.794869 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.797399 25509 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:08.797808 25640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:08.842676 27063 consensus_queue.cc:237] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } }
I20240501 14:09:08.843220 28438 raft_consensus.cc:2932] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER c6717727de344c0b90dd50fd5e70883d (127.22.252.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } } }
I20240501 14:09:08.845281 25378 catalog_manager.cc:5484] T a58883c2a8074e01b2170310e3008e52 P 0e3ebe10573a4863b3734daf8f3c4382 reported cstate change: config changed from index 4 to 6, VOTER c6717727de344c0b90dd50fd5e70883d (127.22.252.130) evicted. New cstate: current_term: 2 leader_uuid: "0e3ebe10573a4863b3734daf8f3c4382" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0e3ebe10573a4863b3734daf8f3c4382" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 42583 } } }
I20240501 14:09:08.847842 25640 tablet_service.cc:1514] Processing DeleteTablet for tablet a58883c2a8074e01b2170310e3008e52 with delete_type TABLET_DATA_TOMBSTONED (TS c6717727de344c0b90dd50fd5e70883d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60440
I20240501 14:09:08.848081 28430 tablet_replica.cc:330] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d: stopping tablet replica
I20240501 14:09:08.848212 28430 raft_consensus.cc:2229] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:08.848361 28430 raft_consensus.cc:2258] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:08.848917 28430 ts_tablet_manager.cc:1875] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:08.850724 28430 ts_tablet_manager.cc:1888] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 14:09:08.850793 28430 log.cc:1196] T a58883c2a8074e01b2170310e3008e52 P c6717727de344c0b90dd50fd5e70883d: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/a58883c2a8074e01b2170310e3008e52
I20240501 14:09:08.851297 25373 catalog_manager.cc:4830] TS c6717727de344c0b90dd50fd5e70883d (127.22.252.130:44715): tablet a58883c2a8074e01b2170310e3008e52 (table rebalance_test_table_3 [id=af481f6478684cf7ab01dc10b39463a7]) successfully deleted
I20240501 14:09:08.899276 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.901228 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.901858 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.902331 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.902771 25640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:08.913867 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.935693 25509 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:08.967716 26792 consensus_queue.cc:237] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "86b991e2c8414b6698e51851d34bc8f2" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 37205 } }
I20240501 14:09:08.968477 28756 raft_consensus.cc:2932] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER e8f33be94f864561bec2e185b5a695c3 (127.22.252.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "86b991e2c8414b6698e51851d34bc8f2" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 37205 } } }
I20240501 14:09:08.970598 25378 catalog_manager.cc:5484] T 8a1f7cc1afb143a8a132944682f969fe P 86b991e2c8414b6698e51851d34bc8f2 reported cstate change: config changed from index 4 to 6, VOTER e8f33be94f864561bec2e185b5a695c3 (127.22.252.129) evicted. New cstate: current_term: 2 leader_uuid: "86b991e2c8414b6698e51851d34bc8f2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "86b991e2c8414b6698e51851d34bc8f2" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 37205 } } }
I20240501 14:09:08.973028 25509 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a1f7cc1afb143a8a132944682f969fe with delete_type TABLET_DATA_TOMBSTONED (TS e8f33be94f864561bec2e185b5a695c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44734
I20240501 14:09:08.973256 28488 tablet_replica.cc:330] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3: stopping tablet replica
I20240501 14:09:08.973423 28488 raft_consensus.cc:2229] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:08.973551 28488 raft_consensus.cc:2258] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:08.974274 28488 ts_tablet_manager.cc:1875] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:08.976339 28488 ts_tablet_manager.cc:1888] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 14:09:08.976409 28488 log.cc:1196] T 8a1f7cc1afb143a8a132944682f969fe P e8f33be94f864561bec2e185b5a695c3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/8a1f7cc1afb143a8a132944682f969fe
I20240501 14:09:08.976886 25374 catalog_manager.cc:4830] TS e8f33be94f864561bec2e185b5a695c3 (127.22.252.129:44353): tablet 8a1f7cc1afb143a8a132944682f969fe (table rebalance_test_table_0 [id=aca60ec23c5145babbcbeaaa484c4348]) successfully deleted
I20240501 14:09:09.030333 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.032124 25640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:09.032341 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.033335 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.038133 25509 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:09.054828 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.057476 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.095158 26660 consensus_queue.cc:237] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } }
I20240501 14:09:09.095714 28496 raft_consensus.cc:2932] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER c6717727de344c0b90dd50fd5e70883d (127.22.252.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } } }
I20240501 14:09:09.097929 25379 catalog_manager.cc:5484] T 4e9029b14eda48cab26e246ed7535d65 P 8a9c183caefa4437b53752c1203b8bb7 reported cstate change: config changed from index 4 to 6, VOTER c6717727de344c0b90dd50fd5e70883d (127.22.252.130) evicted. New cstate: current_term: 2 leader_uuid: "8a9c183caefa4437b53752c1203b8bb7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8a9c183caefa4437b53752c1203b8bb7" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 36947 } } }
I20240501 14:09:09.100296 25640 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e9029b14eda48cab26e246ed7535d65 with delete_type TABLET_DATA_TOMBSTONED (TS c6717727de344c0b90dd50fd5e70883d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60440
I20240501 14:09:09.100512 28430 tablet_replica.cc:330] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d: stopping tablet replica
I20240501 14:09:09.100653 28430 raft_consensus.cc:2229] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:09.100764 28430 raft_consensus.cc:2258] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:09.101380 28430 ts_tablet_manager.cc:1875] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:09.103145 28430 ts_tablet_manager.cc:1888] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 14:09:09.103230 28430 log.cc:1196] T 4e9029b14eda48cab26e246ed7535d65 P c6717727de344c0b90dd50fd5e70883d: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/4e9029b14eda48cab26e246ed7535d65
I20240501 14:09:09.103745 25373 catalog_manager.cc:4830] TS c6717727de344c0b90dd50fd5e70883d (127.22.252.130:44715): tablet 4e9029b14eda48cab26e246ed7535d65 (table rebalance_test_table_1 [id=7f86e635baf845c191b673d7a95b523a]) successfully deleted
I20240501 14:09:09.151999 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.153910 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.157781 25640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:09.162050 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.165906 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.168291 25509 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:09.177306 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.215665 26932 consensus_queue.cc:237] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } }
I20240501 14:09:09.216274 28255 raft_consensus.cc:2932] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER e8f33be94f864561bec2e185b5a695c3 (127.22.252.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } } }
I20240501 14:09:09.218449 25387 catalog_manager.cc:5484] T c230502585ed4dd1aa75bc02a3e7d242 P e9e15d9915264a62a5c9e68f67872199 reported cstate change: config changed from index 4 to 6, VOTER e8f33be94f864561bec2e185b5a695c3 (127.22.252.129) evicted. New cstate: current_term: 2 leader_uuid: "e9e15d9915264a62a5c9e68f67872199" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e9e15d9915264a62a5c9e68f67872199" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 42377 } } }
I20240501 14:09:09.220775 25509 tablet_service.cc:1514] Processing DeleteTablet for tablet c230502585ed4dd1aa75bc02a3e7d242 with delete_type TABLET_DATA_TOMBSTONED (TS e8f33be94f864561bec2e185b5a695c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44734
I20240501 14:09:09.220991 28488 tablet_replica.cc:330] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3: stopping tablet replica
I20240501 14:09:09.221101 28488 raft_consensus.cc:2229] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:09.221197 28488 raft_consensus.cc:2258] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:09.221812 28488 ts_tablet_manager.cc:1875] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:09.223807 28488 ts_tablet_manager.cc:1888] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 14:09:09.223883 28488 log.cc:1196] T c230502585ed4dd1aa75bc02a3e7d242 P e8f33be94f864561bec2e185b5a695c3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/c230502585ed4dd1aa75bc02a3e7d242
I20240501 14:09:09.224431 25374 catalog_manager.cc:4830] TS e8f33be94f864561bec2e185b5a695c3 (127.22.252.129:44353): tablet c230502585ed4dd1aa75bc02a3e7d242 (table rebalance_test_table_2 [id=ec7f7fdeebf748848f7e012922717645]) successfully deleted
I20240501 14:09:09.517190 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.521281 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.523334 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.523615 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.527033 25640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:09.527806 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.529798 25509 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:09.619403 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.619567 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.621670 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.622737 25509 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:09.624711 25640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:09.626317 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.646466 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.859448 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.861024 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.864847 25509 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:09.873448 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.874423 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.876413 25640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:09.876960 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.995967 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.004941 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.006703 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.006954 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.009486 25640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:10.011579 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.016670 25509 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:10.117074 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.117305 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.118304 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.121301 25509 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:10.130381 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.133299 25640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:10.145274 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ee2720787e344e1a83ff5dc97e1e323e | 127.22.252.190:44241 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.22.252.148:35091 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
0e3ebe10573a4863b3734daf8f3c4382 | 127.22.252.134:42583 | HEALTHY | <none> | 2 | 0
3df2596a705349f185baa3b69dabfc25 | 127.22.252.135:39631 | HEALTHY | <none> | 2 | 0
86b991e2c8414b6698e51851d34bc8f2 | 127.22.252.132:37205 | HEALTHY | <none> | 2 | 0
8a9c183caefa4437b53752c1203b8bb7 | 127.22.252.131:36947 | HEALTHY | <none> | 2 | 0
c6717727de344c0b90dd50fd5e70883d | 127.22.252.130:44715 | HEALTHY | <none> | 3 | 0
e8f33be94f864561bec2e185b5a695c3 | 127.22.252.129:44353 | HEALTHY | <none> | 3 | 0
e9e15d9915264a62a5c9e68f67872199 | 127.22.252.133:42377 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.22.252.129 | experimental | 127.22.252.129:44353
local_ip_for_outbound_sockets | 127.22.252.130 | experimental | 127.22.252.130:44715
local_ip_for_outbound_sockets | 127.22.252.131 | experimental | 127.22.252.131:36947
local_ip_for_outbound_sockets | 127.22.252.132 | experimental | 127.22.252.132:37205
local_ip_for_outbound_sockets | 127.22.252.133 | experimental | 127.22.252.133:42377
local_ip_for_outbound_sockets | 127.22.252.134 | experimental | 127.22.252.134:42583
local_ip_for_outbound_sockets | 127.22.252.135 | experimental | 127.22.252.135:39631
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.22.252.129:44353
server_dump_info_path | /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.252.130:44715
server_dump_info_path | /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.252.131:36947
server_dump_info_path | /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.22.252.132:37205
server_dump_info_path | /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.252.133:42377
server_dump_info_path | /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.252.134:42583
server_dump_info_path | /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.22.252.135:39631
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.22.252.148:35091 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:09:10.169034 23538 log_verifier.cc:126] Checking tablet 36b4476375e546298f3d8184dda64204
I20240501 14:09:10.180155 23538 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 36b4476375e546298f3d8184dda64204
I20240501 14:09:10.180262 23538 log_verifier.cc:126] Checking tablet 4ae562f0735b4afeb52172957637cbcf
I20240501 14:09:10.187995 23538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4ae562f0735b4afeb52172957637cbcf
I20240501 14:09:10.188045 23538 log_verifier.cc:126] Checking tablet 4dfd32332f154be489d3a15a2192d07a
I20240501 14:09:10.196221 23538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4dfd32332f154be489d3a15a2192d07a
I20240501 14:09:10.196311 23538 log_verifier.cc:126] Checking tablet 4e9029b14eda48cab26e246ed7535d65
I20240501 14:09:10.204841 23538 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 4e9029b14eda48cab26e246ed7535d65
I20240501 14:09:10.204893 23538 log_verifier.cc:126] Checking tablet 50a6b368aa5e414c9a6464b99f5ea51f
I20240501 14:09:10.213893 23538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 50a6b368aa5e414c9a6464b99f5ea51f
I20240501 14:09:10.213951 23538 log_verifier.cc:126] Checking tablet 78ff5bcb4c5445659d365e84dc810ebc
I20240501 14:09:10.224150 23538 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 78ff5bcb4c5445659d365e84dc810ebc
I20240501 14:09:10.224213 23538 log_verifier.cc:126] Checking tablet 7d50344b10764e408b77e24aad669654
I20240501 14:09:10.232609 23538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7d50344b10764e408b77e24aad669654
I20240501 14:09:10.232656 23538 log_verifier.cc:126] Checking tablet 82731ca00eea49248cf51765e441fea6
I20240501 14:09:10.241046 23538 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 82731ca00eea49248cf51765e441fea6
I20240501 14:09:10.241123 23538 log_verifier.cc:126] Checking tablet 844ba8b3ff6c4d88a101d1c38e7ad0b3
I20240501 14:09:10.248433 23538 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 844ba8b3ff6c4d88a101d1c38e7ad0b3
I20240501 14:09:10.248500 23538 log_verifier.cc:126] Checking tablet 8a1f7cc1afb143a8a132944682f969fe
I20240501 14:09:10.257267 23538 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 8a1f7cc1afb143a8a132944682f969fe
I20240501 14:09:10.257321 23538 log_verifier.cc:126] Checking tablet 90021b467e724c03850f3718c014742e
I20240501 14:09:10.265633 23538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 90021b467e724c03850f3718c014742e
I20240501 14:09:10.265687 23538 log_verifier.cc:126] Checking tablet 9bc7d5eb523e4748b826531bc6be85ac
I20240501 14:09:10.273972 23538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9bc7d5eb523e4748b826531bc6be85ac
I20240501 14:09:10.274029 23538 log_verifier.cc:126] Checking tablet a58883c2a8074e01b2170310e3008e52
I20240501 14:09:10.282274 23538 log_verifier.cc:177] Verified matching terms for 6 ops in tablet a58883c2a8074e01b2170310e3008e52
I20240501 14:09:10.282328 23538 log_verifier.cc:126] Checking tablet c230502585ed4dd1aa75bc02a3e7d242
I20240501 14:09:10.290587 23538 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c230502585ed4dd1aa75bc02a3e7d242
I20240501 14:09:10.290639 23538 log_verifier.cc:126] Checking tablet d16f3d9f72b44c2cb15a6c634bee2a28
I20240501 14:09:10.298828 23538 log_verifier.cc:177] Verified matching terms for 6 ops in tablet d16f3d9f72b44c2cb15a6c634bee2a28
I20240501 14:09:10.298879 23538 log_verifier.cc:126] Checking tablet fd9a83b1cc1441da972fb41466aaf402
I20240501 14:09:10.307166 23538 log_verifier.cc:177] Verified matching terms for 6 ops in tablet fd9a83b1cc1441da972fb41466aaf402
I20240501 14:09:10.309260 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.136:0
--local_ip_for_outbound_sockets=127.22.252.136
--webserver_interface=127.22.252.136
--webserver_port=0
--tserver_master_addrs=127.22.252.190:44241
--builtin_ntp_servers=127.22.252.148:35091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:09:10.415413 29081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:10.415591 29081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:10.415627 29081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:10.418475 29081 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:10.418531 29081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:10.418593 29081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.136
I20240501 14:09:10.422022 29081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:35091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.252.136
--webserver_port=0
--tserver_master_addrs=127.22.252.190:44241
--never_fsync=true
--heap_profile_path=/tmp/kudu.29081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.136
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:10.422761 29081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:10.423772 29081 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:10.432418 29086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:10.433082 29087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:10.434010 29081 server_base.cc:1034] running on GCE node
W20240501 14:09:10.434692 29089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:10.435015 29081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:10.435516 29081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:10.436669 29081 hybrid_clock.cc:648] HybridClock initialized: now 1714572550436655 us; error 45 us; skew 500 ppm
I20240501 14:09:10.438333 29081 webserver.cc:460] Webserver started at http://127.22.252.136:33433/ using document root <none> and password file <none>
I20240501 14:09:10.438843 29081 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:10.438901 29081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:10.439118 29081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:10.440724 29081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "a541f200ec6340ef974d45176fe1b198"
format_stamp: "Formatted at 2024-05-01 14:09:10 on dist-test-slave-bkzb"
I20240501 14:09:10.441205 29081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "a541f200ec6340ef974d45176fe1b198"
format_stamp: "Formatted at 2024-05-01 14:09:10 on dist-test-slave-bkzb"
I20240501 14:09:10.444334 29081 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:09:10.446367 29095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:10.447407 29081 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:10.447505 29081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "a541f200ec6340ef974d45176fe1b198"
format_stamp: "Formatted at 2024-05-01 14:09:10 on dist-test-slave-bkzb"
I20240501 14:09:10.447613 29081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:10.458046 29081 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:10.458566 29081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:10.458766 29081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:10.459245 29081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:10.460022 29081 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:10.460072 29081 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:10.460115 29081 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:10.460162 29081 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:10.467840 29081 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.136:33959
I20240501 14:09:10.467867 29208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.136:33959 every 8 connection(s)
I20240501 14:09:10.468683 29081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:09:10.474815 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 29081
I20240501 14:09:10.474947 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 14:09:10.476660 29209 heartbeater.cc:347] Connected to a master server at 127.22.252.190:44241
I20240501 14:09:10.476907 29209 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:10.477296 29209 heartbeater.cc:511] Master 127.22.252.190:44241 requested a full tablet report, sending...
I20240501 14:09:10.478119 25387 ts_manager.cc:194] Registered new tserver with Master: a541f200ec6340ef974d45176fe1b198 (127.22.252.136:33959)
I20240501 14:09:10.478798 25387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.136:41129
I20240501 14:09:10.663385 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.663609 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.666505 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.673856 25640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:10.679059 25509 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:10.680675 29143 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:10.695300 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.703696 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.805054 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.806365 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.810724 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.819963 29143 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:10.820413 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.821364 25640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:10.829515 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.837954 25509 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:10.879472 25659 consensus_queue.cc:237] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } }
W20240501 14:09:10.883728 25594 consensus_peers.cc:485] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d -> Peer a541f200ec6340ef974d45176fe1b198 (127.22.252.136:33959): Couldn't send request to peer a541f200ec6340ef974d45176fe1b198. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4dfd32332f154be489d3a15a2192d07a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:11.376415 29314 ts_tablet_manager.cc:902] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198: Initiating tablet copy from peer c6717727de344c0b90dd50fd5e70883d (127.22.252.130:44715)
I20240501 14:09:11.377146 29314 tablet_copy_client.cc:308] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.130:44715
I20240501 14:09:11.380597 25680 tablet_copy_service.cc:140] P c6717727de344c0b90dd50fd5e70883d: Received BeginTabletCopySession request for tablet 4dfd32332f154be489d3a15a2192d07a from peer a541f200ec6340ef974d45176fe1b198 ({username='slave'} at 127.22.252.136:48129)
I20240501 14:09:11.380682 25680 tablet_copy_service.cc:161] P c6717727de344c0b90dd50fd5e70883d: Beginning new tablet copy session on tablet 4dfd32332f154be489d3a15a2192d07a from peer a541f200ec6340ef974d45176fe1b198 at {username='slave'} at 127.22.252.136:48129: session id = a541f200ec6340ef974d45176fe1b198-4dfd32332f154be489d3a15a2192d07a
I20240501 14:09:11.381348 25680 tablet_copy_source_session.cc:205] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:11.384270 29314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dfd32332f154be489d3a15a2192d07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:11.388789 29314 tablet_copy_client.cc:783] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:11.388957 29314 tablet_copy_client.cc:647] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:11.390273 29314 tablet_copy_client.cc:523] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:11.391577 29314 tablet_bootstrap.cc:492] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198: Bootstrap starting.
I20240501 14:09:11.396309 29314 log.cc:824] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:11.413408 29314 tablet_bootstrap.cc:492] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:09:11.413739 29314 tablet_bootstrap.cc:492] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198: Bootstrap complete.
I20240501 14:09:11.414011 29314 ts_tablet_manager.cc:1367] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.003s
I20240501 14:09:11.415730 29314 raft_consensus.cc:348] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } }
I20240501 14:09:11.416294 29314 raft_consensus.cc:729] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a541f200ec6340ef974d45176fe1b198, State: Initialized, Role: FOLLOWER
I20240501 14:09:11.416575 29314 consensus_queue.cc:260] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } }
I20240501 14:09:11.417179 29209 heartbeater.cc:503] Master 127.22.252.190:44241 was elected leader, sending a full tablet report...
I20240501 14:09:11.417305 29314 ts_tablet_manager.cc:1398] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240501 14:09:11.417930 25680 tablet_copy_service.cc:342] P c6717727de344c0b90dd50fd5e70883d: Request end of tablet copy session a541f200ec6340ef974d45176fe1b198-4dfd32332f154be489d3a15a2192d07a received from {username='slave'} at 127.22.252.136:48129
I20240501 14:09:11.418023 25680 tablet_copy_service.cc:433] P c6717727de344c0b90dd50fd5e70883d: ending tablet copy session a541f200ec6340ef974d45176fe1b198-4dfd32332f154be489d3a15a2192d07a on tablet 4dfd32332f154be489d3a15a2192d07a with peer a541f200ec6340ef974d45176fe1b198
I20240501 14:09:11.845870 29163 raft_consensus.cc:1203] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:11.847045 29322 raft_consensus.cc:2932] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a541f200ec6340ef974d45176fe1b198 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } } }
I20240501 14:09:11.847637 29163 raft_consensus.cc:2932] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a541f200ec6340ef974d45176fe1b198 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } } }
I20240501 14:09:11.850199 25387 catalog_manager.cc:5484] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d reported cstate change: config changed from index -1 to 4, VOTER a541f200ec6340ef974d45176fe1b198 (127.22.252.136) added. New cstate: current_term: 1 leader_uuid: "c6717727de344c0b90dd50fd5e70883d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } } }
I20240501 14:09:11.866844 25528 consensus_queue.cc:237] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } }
W20240501 14:09:11.870779 25463 consensus_peers.cc:485] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 -> Peer a541f200ec6340ef974d45176fe1b198 (127.22.252.136:33959): Couldn't send request to peer a541f200ec6340ef974d45176fe1b198. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ae562f0735b4afeb52172957637cbcf. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:12.445642 29350 ts_tablet_manager.cc:902] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198: Initiating tablet copy from peer e8f33be94f864561bec2e185b5a695c3 (127.22.252.129:44353)
I20240501 14:09:12.446110 29350 tablet_copy_client.cc:308] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.129:44353
I20240501 14:09:12.449538 25549 tablet_copy_service.cc:140] P e8f33be94f864561bec2e185b5a695c3: Received BeginTabletCopySession request for tablet 4ae562f0735b4afeb52172957637cbcf from peer a541f200ec6340ef974d45176fe1b198 ({username='slave'} at 127.22.252.136:56079)
I20240501 14:09:12.449651 25549 tablet_copy_service.cc:161] P e8f33be94f864561bec2e185b5a695c3: Beginning new tablet copy session on tablet 4ae562f0735b4afeb52172957637cbcf from peer a541f200ec6340ef974d45176fe1b198 at {username='slave'} at 127.22.252.136:56079: session id = a541f200ec6340ef974d45176fe1b198-4ae562f0735b4afeb52172957637cbcf
I20240501 14:09:12.450433 25549 tablet_copy_source_session.cc:205] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:12.451325 29350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae562f0735b4afeb52172957637cbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:12.453321 29350 tablet_copy_client.cc:783] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:12.453459 29350 tablet_copy_client.cc:647] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:12.454470 29350 tablet_copy_client.cc:523] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:12.455356 29350 tablet_bootstrap.cc:492] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198: Bootstrap starting.
I20240501 14:09:12.471712 29350 tablet_bootstrap.cc:492] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:09:12.471956 29350 tablet_bootstrap.cc:492] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198: Bootstrap complete.
I20240501 14:09:12.472115 29350 ts_tablet_manager.cc:1367] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20240501 14:09:12.472613 29350 raft_consensus.cc:348] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } }
I20240501 14:09:12.472818 29350 raft_consensus.cc:729] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a541f200ec6340ef974d45176fe1b198, State: Initialized, Role: FOLLOWER
I20240501 14:09:12.472929 29350 consensus_queue.cc:260] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } }
I20240501 14:09:12.473232 29350 ts_tablet_manager.cc:1398] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240501 14:09:12.473733 25549 tablet_copy_service.cc:342] P e8f33be94f864561bec2e185b5a695c3: Request end of tablet copy session a541f200ec6340ef974d45176fe1b198-4ae562f0735b4afeb52172957637cbcf received from {username='slave'} at 127.22.252.136:56079
I20240501 14:09:12.473825 25549 tablet_copy_service.cc:433] P e8f33be94f864561bec2e185b5a695c3: ending tablet copy session a541f200ec6340ef974d45176fe1b198-4ae562f0735b4afeb52172957637cbcf on tablet 4ae562f0735b4afeb52172957637cbcf with peer a541f200ec6340ef974d45176fe1b198
I20240501 14:09:12.907853 29163 raft_consensus.cc:1203] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:12.908788 29360 raft_consensus.cc:2932] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a541f200ec6340ef974d45176fe1b198 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } } }
I20240501 14:09:12.909232 29163 raft_consensus.cc:2932] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a541f200ec6340ef974d45176fe1b198 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } } }
I20240501 14:09:12.910310 29360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:51706 (request call id 0) took 1043 ms. Trace:
I20240501 14:09:12.910427 29360 rpcz_store.cc:276] 0501 14:09:11.866516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 14:09:11.866568 (+ 52us) service_pool.cc:225] Handling call
0501 14:09:11.867270 (+ 702us) log.cc:842] Serialized 263 byte log entry
0501 14:09:12.910288 (+1043018us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":47,"raft.queue_time_us":105,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":122,"spinlock_wait_cycles":3456,"thread_start_us":95,"threads_started":2,"wal-append.queue_time_us":122,"wal-append.run_cpu_time_us":277,"wal-append.run_wall_time_us":1000338}
I20240501 14:09:12.911631 25378 catalog_manager.cc:5484] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198 reported cstate change: config changed from index -1 to 4, VOTER a541f200ec6340ef974d45176fe1b198 (127.22.252.136) added. New cstate: current_term: 1 leader_uuid: "e8f33be94f864561bec2e185b5a695c3" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } } }
I20240501 14:09:12.969677 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:12.969894 29143 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:12.974447 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:12.975905 25509 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:12.985800 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:12.989642 25640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:12.992028 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.013047 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.035542 25528 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4ae562f0735b4afeb52172957637cbcf"
dest_uuid: "e8f33be94f864561bec2e185b5a695c3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51726
I20240501 14:09:13.035696 25528 raft_consensus.cc:595] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:13.087652 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.089161 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.092650 25640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:13.093840 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.095817 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.096743 29143 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:13.102802 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.116568 25509 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:13.200500 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.202631 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.205287 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.205528 29143 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:13.218370 25640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:13.219067 25509 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:13.232511 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.241217 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.264225 25659 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4dfd32332f154be489d3a15a2192d07a"
dest_uuid: "c6717727de344c0b90dd50fd5e70883d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57316
I20240501 14:09:13.264398 25659 raft_consensus.cc:595] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:13.314361 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.319156 29143 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:13.321146 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.324752 25640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:13.327762 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.328330 25509 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:13.332733 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.362008 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.399791 29527 raft_consensus.cc:979] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d: : Instructing follower a541f200ec6340ef974d45176fe1b198 to start an election
I20240501 14:09:13.399951 29528 raft_consensus.cc:1067] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Signalling peer a541f200ec6340ef974d45176fe1b198 to start an election
I20240501 14:09:13.400458 29163 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4dfd32332f154be489d3a15a2192d07a"
dest_uuid: "a541f200ec6340ef974d45176fe1b198"
from {username='slave'} at 127.22.252.130:58645
I20240501 14:09:13.400627 29163 raft_consensus.cc:482] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:13.400707 29163 raft_consensus.cc:3037] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:13.401489 29163 raft_consensus.cc:504] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } }
I20240501 14:09:13.402276 29163 leader_election.cc:288] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198 [CANDIDATE]: Term 2 election: Requested vote from peers c6717727de344c0b90dd50fd5e70883d (127.22.252.130:44715)
I20240501 14:09:13.402654 25659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dfd32332f154be489d3a15a2192d07a" candidate_uuid: "a541f200ec6340ef974d45176fe1b198" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c6717727de344c0b90dd50fd5e70883d"
I20240501 14:09:13.402758 25659 raft_consensus.cc:3032] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:13.402801 25659 raft_consensus.cc:729] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c6717727de344c0b90dd50fd5e70883d, State: Running, Role: LEADER
I20240501 14:09:13.402931 25659 consensus_queue.cc:260] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } }
I20240501 14:09:13.403069 25659 raft_consensus.cc:3037] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:13.403848 25659 raft_consensus.cc:2454] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a541f200ec6340ef974d45176fe1b198 in term 2.
I20240501 14:09:13.404363 29098 leader_election.cc:302] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a541f200ec6340ef974d45176fe1b198, c6717727de344c0b90dd50fd5e70883d; no voters:
I20240501 14:09:13.404582 29358 raft_consensus.cc:2781] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:13.404757 29358 raft_consensus.cc:686] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198 [term 2 LEADER]: Becoming Leader. State: Replica: a541f200ec6340ef974d45176fe1b198, State: Running, Role: LEADER
I20240501 14:09:13.405040 29358 consensus_queue.cc:237] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } }
I20240501 14:09:13.406973 25378 catalog_manager.cc:5484] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198 reported cstate change: term changed from 1 to 2, leader changed from c6717727de344c0b90dd50fd5e70883d (127.22.252.130) to a541f200ec6340ef974d45176fe1b198 (127.22.252.136). New cstate: current_term: 2 leader_uuid: "a541f200ec6340ef974d45176fe1b198" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 } } peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } } }
I20240501 14:09:13.453433 29531 raft_consensus.cc:979] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3: : Instructing follower a541f200ec6340ef974d45176fe1b198 to start an election
I20240501 14:09:13.453528 29531 raft_consensus.cc:1067] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Signalling peer a541f200ec6340ef974d45176fe1b198 to start an election
I20240501 14:09:13.453929 29163 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ae562f0735b4afeb52172957637cbcf"
dest_uuid: "a541f200ec6340ef974d45176fe1b198"
from {username='slave'} at 127.22.252.129:45011
I20240501 14:09:13.454067 29163 raft_consensus.cc:482] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:13.454110 29163 raft_consensus.cc:3037] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:13.454874 29163 raft_consensus.cc:504] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } }
I20240501 14:09:13.455226 29163 leader_election.cc:288] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198 [CANDIDATE]: Term 2 election: Requested vote from peers e8f33be94f864561bec2e185b5a695c3 (127.22.252.129:44353)
I20240501 14:09:13.455610 25528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae562f0735b4afeb52172957637cbcf" candidate_uuid: "a541f200ec6340ef974d45176fe1b198" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e8f33be94f864561bec2e185b5a695c3"
I20240501 14:09:13.455724 25528 raft_consensus.cc:3032] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:13.455770 25528 raft_consensus.cc:729] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8f33be94f864561bec2e185b5a695c3, State: Running, Role: LEADER
I20240501 14:09:13.455938 25528 consensus_queue.cc:260] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } }
I20240501 14:09:13.456068 25528 raft_consensus.cc:3037] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:13.456841 25528 raft_consensus.cc:2454] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a541f200ec6340ef974d45176fe1b198 in term 2.
I20240501 14:09:13.457127 29099 leader_election.cc:302] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a541f200ec6340ef974d45176fe1b198, e8f33be94f864561bec2e185b5a695c3; no voters:
I20240501 14:09:13.457327 29358 raft_consensus.cc:2781] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:13.457423 29358 raft_consensus.cc:686] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198 [term 2 LEADER]: Becoming Leader. State: Replica: a541f200ec6340ef974d45176fe1b198, State: Running, Role: LEADER
I20240501 14:09:13.457556 29358 consensus_queue.cc:237] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } }
I20240501 14:09:13.459076 25378 catalog_manager.cc:5484] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198 reported cstate change: term changed from 1 to 2, leader changed from e8f33be94f864561bec2e185b5a695c3 (127.22.252.129) to a541f200ec6340ef974d45176fe1b198 (127.22.252.136). New cstate: current_term: 2 leader_uuid: "a541f200ec6340ef974d45176fe1b198" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 } } peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } } }
I20240501 14:09:13.637643 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.638170 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.643100 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.651415 25640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.651448 29143 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.652587 25509 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.661942 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.664669 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.741993 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.746194 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.746785 29143 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.750368 25640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.756379 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.757720 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.769822 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.782550 25509 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.913391 25659 raft_consensus.cc:1261] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [term 2 FOLLOWER]: Refusing update from remote peer a541f200ec6340ef974d45176fe1b198: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:13.913921 29358 consensus_queue.cc:1035] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6717727de344c0b90dd50fd5e70883d" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 44715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:13.932621 25528 raft_consensus.cc:1261] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Refusing update from remote peer a541f200ec6340ef974d45176fe1b198: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:13.933058 29529 consensus_queue.cc:1035] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8f33be94f864561bec2e185b5a695c3" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 44353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:14.055459 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.062849 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.063758 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.064949 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.070319 29143 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.081344 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.082633 25640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.091428 25509 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.127450 29163 consensus_queue.cc:237] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } }
I20240501 14:09:14.128065 29358 raft_consensus.cc:2932] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER e8f33be94f864561bec2e185b5a695c3 (127.22.252.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } } }
I20240501 14:09:14.130307 25387 catalog_manager.cc:5484] T 4ae562f0735b4afeb52172957637cbcf P a541f200ec6340ef974d45176fe1b198 reported cstate change: config changed from index 4 to 6, VOTER e8f33be94f864561bec2e185b5a695c3 (127.22.252.129) evicted. New cstate: current_term: 2 leader_uuid: "a541f200ec6340ef974d45176fe1b198" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } } }
I20240501 14:09:14.133052 25509 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ae562f0735b4afeb52172957637cbcf with delete_type TABLET_DATA_TOMBSTONED (TS e8f33be94f864561bec2e185b5a695c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44734
I20240501 14:09:14.133433 29676 tablet_replica.cc:330] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3: stopping tablet replica
I20240501 14:09:14.133576 29676 raft_consensus.cc:2229] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:14.133705 29676 raft_consensus.cc:2258] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:14.135037 29676 ts_tablet_manager.cc:1875] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:14.137153 29676 ts_tablet_manager.cc:1888] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 14:09:14.137220 29676 log.cc:1196] T 4ae562f0735b4afeb52172957637cbcf P e8f33be94f864561bec2e185b5a695c3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/4ae562f0735b4afeb52172957637cbcf
I20240501 14:09:14.137732 25374 catalog_manager.cc:4830] TS e8f33be94f864561bec2e185b5a695c3 (127.22.252.129:44353): tablet 4ae562f0735b4afeb52172957637cbcf (table rebalance_test_table_3 [id=af481f6478684cf7ab01dc10b39463a7]) successfully deleted
I20240501 14:09:14.186424 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.188550 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.194061 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.195065 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.201540 25509 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.207964 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.214599 29143 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.217077 25640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.258805 29163 consensus_queue.cc:237] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } }
I20240501 14:09:14.259382 29358 raft_consensus.cc:2932] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER c6717727de344c0b90dd50fd5e70883d (127.22.252.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } } }
I20240501 14:09:14.261420 25378 catalog_manager.cc:5484] T 4dfd32332f154be489d3a15a2192d07a P a541f200ec6340ef974d45176fe1b198 reported cstate change: config changed from index 4 to 6, VOTER c6717727de344c0b90dd50fd5e70883d (127.22.252.130) evicted. New cstate: current_term: 2 leader_uuid: "a541f200ec6340ef974d45176fe1b198" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a541f200ec6340ef974d45176fe1b198" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 33959 } } }
I20240501 14:09:14.263656 25640 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dfd32332f154be489d3a15a2192d07a with delete_type TABLET_DATA_TOMBSTONED (TS c6717727de344c0b90dd50fd5e70883d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60440
I20240501 14:09:14.263970 29725 tablet_replica.cc:330] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d: stopping tablet replica
I20240501 14:09:14.264092 29725 raft_consensus.cc:2229] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:14.264195 29725 raft_consensus.cc:2258] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:14.265472 29725 ts_tablet_manager.cc:1875] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:14.267192 29725 ts_tablet_manager.cc:1888] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 14:09:14.267261 29725 log.cc:1196] T 4dfd32332f154be489d3a15a2192d07a P c6717727de344c0b90dd50fd5e70883d: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/4dfd32332f154be489d3a15a2192d07a
I20240501 14:09:14.267747 25373 catalog_manager.cc:4830] TS c6717727de344c0b90dd50fd5e70883d (127.22.252.130:44715): tablet 4dfd32332f154be489d3a15a2192d07a (table rebalance_test_table_4 [id=0622125ee1f04076a30c4aa37417bf64]) successfully deleted
I20240501 14:09:14.532510 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.534735 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.542423 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.542740 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.543982 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.546308 25640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.549172 25509 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.563724 29143 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.634244 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.639398 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.642292 25640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.644599 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.645699 29143 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.662067 25509 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.673377 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.673836 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.756055 27041 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.758030 26772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.759268 29143 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.761205 27174 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.773572 26640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.779080 26912 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.790030 25509 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.794178 25640 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ee2720787e344e1a83ff5dc97e1e323e | 127.22.252.190:44241 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.22.252.148:35091 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
0e3ebe10573a4863b3734daf8f3c4382 | 127.22.252.134:42583 | HEALTHY | <none> | 2 | 0
3df2596a705349f185baa3b69dabfc25 | 127.22.252.135:39631 | HEALTHY | <none> | 2 | 0
86b991e2c8414b6698e51851d34bc8f2 | 127.22.252.132:37205 | HEALTHY | <none> | 2 | 0
8a9c183caefa4437b53752c1203b8bb7 | 127.22.252.131:36947 | HEALTHY | <none> | 2 | 0
a541f200ec6340ef974d45176fe1b198 | 127.22.252.136:33959 | HEALTHY | <none> | 2 | 0
c6717727de344c0b90dd50fd5e70883d | 127.22.252.130:44715 | HEALTHY | <none> | 2 | 0
e8f33be94f864561bec2e185b5a695c3 | 127.22.252.129:44353 | HEALTHY | <none> | 2 | 0
e9e15d9915264a62a5c9e68f67872199 | 127.22.252.133:42377 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.22.252.129 | experimental | 127.22.252.129:44353
local_ip_for_outbound_sockets | 127.22.252.130 | experimental | 127.22.252.130:44715
local_ip_for_outbound_sockets | 127.22.252.131 | experimental | 127.22.252.131:36947
local_ip_for_outbound_sockets | 127.22.252.132 | experimental | 127.22.252.132:37205
local_ip_for_outbound_sockets | 127.22.252.133 | experimental | 127.22.252.133:42377
local_ip_for_outbound_sockets | 127.22.252.134 | experimental | 127.22.252.134:42583
local_ip_for_outbound_sockets | 127.22.252.135 | experimental | 127.22.252.135:39631
local_ip_for_outbound_sockets | 127.22.252.136 | experimental | 127.22.252.136:33959
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.22.252.129:44353
server_dump_info_path | /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.252.130:44715
server_dump_info_path | /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.252.131:36947
server_dump_info_path | /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.22.252.132:37205
server_dump_info_path | /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.252.133:42377
server_dump_info_path | /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.252.134:42583
server_dump_info_path | /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.22.252.135:39631
server_dump_info_path | /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.22.252.136:33959
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.22.252.148:35091 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:09:14.815935 23538 log_verifier.cc:126] Checking tablet 36b4476375e546298f3d8184dda64204
I20240501 14:09:14.823978 23538 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 36b4476375e546298f3d8184dda64204
I20240501 14:09:14.824040 23538 log_verifier.cc:126] Checking tablet 4ae562f0735b4afeb52172957637cbcf
I20240501 14:09:14.834308 23538 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 4ae562f0735b4afeb52172957637cbcf
I20240501 14:09:14.834374 23538 log_verifier.cc:126] Checking tablet 4dfd32332f154be489d3a15a2192d07a
I20240501 14:09:14.842799 23538 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 4dfd32332f154be489d3a15a2192d07a
I20240501 14:09:14.842870 23538 log_verifier.cc:126] Checking tablet 4e9029b14eda48cab26e246ed7535d65
I20240501 14:09:14.848363 23538 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 4e9029b14eda48cab26e246ed7535d65
I20240501 14:09:14.848405 23538 log_verifier.cc:126] Checking tablet 50a6b368aa5e414c9a6464b99f5ea51f
I20240501 14:09:14.853751 23538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 50a6b368aa5e414c9a6464b99f5ea51f
I20240501 14:09:14.853791 23538 log_verifier.cc:126] Checking tablet 78ff5bcb4c5445659d365e84dc810ebc
I20240501 14:09:14.859210 23538 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 78ff5bcb4c5445659d365e84dc810ebc
I20240501 14:09:14.859254 23538 log_verifier.cc:126] Checking tablet 7d50344b10764e408b77e24aad669654
I20240501 14:09:14.864562 23538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7d50344b10764e408b77e24aad669654
I20240501 14:09:14.864598 23538 log_verifier.cc:126] Checking tablet 82731ca00eea49248cf51765e441fea6
I20240501 14:09:14.870033 23538 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 82731ca00eea49248cf51765e441fea6
I20240501 14:09:14.870072 23538 log_verifier.cc:126] Checking tablet 844ba8b3ff6c4d88a101d1c38e7ad0b3
I20240501 14:09:14.874541 23538 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 844ba8b3ff6c4d88a101d1c38e7ad0b3
I20240501 14:09:14.874575 23538 log_verifier.cc:126] Checking tablet 8a1f7cc1afb143a8a132944682f969fe
I20240501 14:09:14.880031 23538 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 8a1f7cc1afb143a8a132944682f969fe
I20240501 14:09:14.880070 23538 log_verifier.cc:126] Checking tablet 90021b467e724c03850f3718c014742e
I20240501 14:09:14.885267 23538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 90021b467e724c03850f3718c014742e
I20240501 14:09:14.885300 23538 log_verifier.cc:126] Checking tablet 9bc7d5eb523e4748b826531bc6be85ac
I20240501 14:09:14.890630 23538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9bc7d5eb523e4748b826531bc6be85ac
I20240501 14:09:14.890667 23538 log_verifier.cc:126] Checking tablet a58883c2a8074e01b2170310e3008e52
I20240501 14:09:14.896154 23538 log_verifier.cc:177] Verified matching terms for 6 ops in tablet a58883c2a8074e01b2170310e3008e52
I20240501 14:09:14.896191 23538 log_verifier.cc:126] Checking tablet c230502585ed4dd1aa75bc02a3e7d242
I20240501 14:09:14.901590 23538 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c230502585ed4dd1aa75bc02a3e7d242
I20240501 14:09:14.901629 23538 log_verifier.cc:126] Checking tablet d16f3d9f72b44c2cb15a6c634bee2a28
I20240501 14:09:14.907178 23538 log_verifier.cc:177] Verified matching terms for 6 ops in tablet d16f3d9f72b44c2cb15a6c634bee2a28
I20240501 14:09:14.907215 23538 log_verifier.cc:126] Checking tablet fd9a83b1cc1441da972fb41466aaf402
I20240501 14:09:14.912526 23538 log_verifier.cc:177] Verified matching terms for 6 ops in tablet fd9a83b1cc1441da972fb41466aaf402
I20240501 14:09:14.912987 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 25447
I20240501 14:09:14.920594 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 25578
I20240501 14:09:14.926738 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 26578
I20240501 14:09:14.932181 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 26709
I20240501 14:09:14.937464 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 26842
I20240501 14:09:14.942760 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 26981
I20240501 14:09:14.948173 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 27111
I20240501 14:09:14.953485 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 29081
I20240501 14:09:14.958856 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 25356
2024-05-01T14:09:14Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/0 (26346 ms)
[----------] 1 test from RebalanceParamTest (26346 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0
I20240501 14:09:14.998330 23538 test_util.cc:268] Using random seed: 135598645
I20240501 14:09:14.999060 23538 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:09:14.999095 23538 ts_itest-base.cc:113] --------------
I20240501 14:09:14.999106 23538 ts_itest-base.cc:114] 8 tablet servers
I20240501 14:09:14.999117 23538 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:09:14.999127 23538 ts_itest-base.cc:116] --------------
2024-05-01T14:09:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:09:15Z Disabled control of system clock
I20240501 14:09:15.007485 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.252.190:35675
--webserver_interface=127.22.252.190
--webserver_port=0
--builtin_ntp_servers=127.22.252.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.252.190:35675
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 14:09:15.109637 29842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:15.109849 29842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:15.109887 29842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:15.112766 29842 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:15.112818 29842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:09:15.112840 29842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:15.112860 29842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:09:15.112882 29842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:09:15.116369 29842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.252.190:35675
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.252.190:35675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.252.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.29842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:15.117128 29842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:15.118168 29842 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:15.125177 29847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:15.126122 29850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:15.126863 29848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:15.127588 29842 server_base.cc:1034] running on GCE node
I20240501 14:09:15.128032 29842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:15.128942 29842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:15.130112 29842 hybrid_clock.cc:648] HybridClock initialized: now 1714572555130091 us; error 39 us; skew 500 ppm
I20240501 14:09:15.135982 29842 webserver.cc:460] Webserver started at http://127.22.252.190:42361/ using document root <none> and password file <none>
I20240501 14:09:15.136548 29842 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:15.136610 29842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:15.136776 29842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:15.138351 29842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f2cd510126964a349f8d7712a2b33cf5"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-bkzb"
I20240501 14:09:15.138792 29842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f2cd510126964a349f8d7712a2b33cf5"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-bkzb"
I20240501 14:09:15.141923 29842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:09:15.144006 29856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.144976 29842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:15.145082 29842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f2cd510126964a349f8d7712a2b33cf5"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-bkzb"
I20240501 14:09:15.145161 29842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:15.172544 29842 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:15.173079 29842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:15.173233 29842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:15.179205 29908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.190:35675 every 8 connection(s)
I20240501 14:09:15.179190 29842 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.190:35675
I20240501 14:09:15.180140 29842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:09:15.182554 29909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:15.182552 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 29842
I20240501 14:09:15.182650 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:09:15.189993 29909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Bootstrap starting.
I20240501 14:09:15.192092 29909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:15.192855 29909 log.cc:824] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:15.194382 29909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: No bootstrap required, opened a new log
I20240501 14:09:15.196760 29909 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2cd510126964a349f8d7712a2b33cf5" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 35675 } }
I20240501 14:09:15.196916 29909 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:15.196955 29909 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2cd510126964a349f8d7712a2b33cf5, State: Initialized, Role: FOLLOWER
I20240501 14:09:15.197337 29909 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2cd510126964a349f8d7712a2b33cf5" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 35675 } }
I20240501 14:09:15.197458 29909 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:15.197516 29909 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:15.197613 29909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:15.198486 29909 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2cd510126964a349f8d7712a2b33cf5" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 35675 } }
I20240501 14:09:15.198752 29909 leader_election.cc:302] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2cd510126964a349f8d7712a2b33cf5; no voters:
I20240501 14:09:15.198954 29909 leader_election.cc:288] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:15.199323 29914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:15.199570 29914 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5 [term 1 LEADER]: Becoming Leader. State: Replica: f2cd510126964a349f8d7712a2b33cf5, State: Running, Role: LEADER
I20240501 14:09:15.199857 29914 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2cd510126964a349f8d7712a2b33cf5" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 35675 } }
I20240501 14:09:15.199932 29909 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:09:15.201284 29915 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2cd510126964a349f8d7712a2b33cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2cd510126964a349f8d7712a2b33cf5" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 35675 } } }
I20240501 14:09:15.201313 29916 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2cd510126964a349f8d7712a2b33cf5. Latest consensus state: current_term: 1 leader_uuid: "f2cd510126964a349f8d7712a2b33cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2cd510126964a349f8d7712a2b33cf5" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 35675 } } }
I20240501 14:09:15.201400 29915 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5 [sys.catalog]: This master's current role is: LEADER
I20240501 14:09:15.201406 29916 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5 [sys.catalog]: This master's current role is: LEADER
I20240501 14:09:15.201900 29924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:09:15.204710 29924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:09:15.209245 29924 catalog_manager.cc:1347] Generated new cluster ID: 007931b1f5b64a429a1011224c061afa
I20240501 14:09:15.209308 29924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:09:15.221761 29924 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:09:15.222877 29924 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:09:15.228778 29924 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Generated new TSK 0
I20240501 14:09:15.229494 29924 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:09:15.242602 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.129:0
--local_ip_for_outbound_sockets=127.22.252.129
--webserver_interface=127.22.252.129
--webserver_port=0
--tserver_master_addrs=127.22.252.190:35675
--builtin_ntp_servers=127.22.252.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:09:15.346344 29933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:15.346551 29933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:15.346593 29933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:15.349684 29933 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:15.349741 29933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:15.349828 29933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.129
I20240501 14:09:15.353271 29933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.252.129
--webserver_port=0
--tserver_master_addrs=127.22.252.190:35675
--never_fsync=true
--heap_profile_path=/tmp/kudu.29933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.129
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:15.354036 29933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:15.355094 29933 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:15.362934 29938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:15.363898 29939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:15.364646 29941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:15.365535 29933 server_base.cc:1034] running on GCE node
I20240501 14:09:15.365850 29933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:15.366338 29933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:15.367499 29933 hybrid_clock.cc:648] HybridClock initialized: now 1714572555367476 us; error 38 us; skew 500 ppm
I20240501 14:09:15.369241 29933 webserver.cc:460] Webserver started at http://127.22.252.129:40527/ using document root <none> and password file <none>
I20240501 14:09:15.369751 29933 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:15.369813 29933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:15.369987 29933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:15.371541 29933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "93f2518fbadf4f9991cda93afacb363e"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-bkzb"
I20240501 14:09:15.372006 29933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "93f2518fbadf4f9991cda93afacb363e"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-bkzb"
I20240501 14:09:15.375263 29933 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:15.377333 29947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.378297 29933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:15.378398 29933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "93f2518fbadf4f9991cda93afacb363e"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-bkzb"
I20240501 14:09:15.378484 29933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:15.396598 29933 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:15.397156 29933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:15.397291 29933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:15.397809 29933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:15.398710 29933 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:15.398759 29933 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.398811 29933 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:15.398847 29933 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.406776 29933 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.129:41387
I20240501 14:09:15.406803 30060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.129:41387 every 8 connection(s)
I20240501 14:09:15.407635 29933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:09:15.408011 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 29933
I20240501 14:09:15.408128 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:09:15.410430 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.130:0
--local_ip_for_outbound_sockets=127.22.252.130
--webserver_interface=127.22.252.130
--webserver_port=0
--tserver_master_addrs=127.22.252.190:35675
--builtin_ntp_servers=127.22.252.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:09:15.416692 30061 heartbeater.cc:347] Connected to a master server at 127.22.252.190:35675
I20240501 14:09:15.416893 30061 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:15.417295 30061 heartbeater.cc:511] Master 127.22.252.190:35675 requested a full tablet report, sending...
I20240501 14:09:15.418578 29873 ts_manager.cc:194] Registered new tserver with Master: 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:15.419673 29873 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.129:57605
W20240501 14:09:15.512727 30064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:15.512931 30064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:15.512979 30064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:15.515910 30064 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:15.515967 30064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:15.516062 30064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.130
I20240501 14:09:15.519572 30064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.252.130
--webserver_port=0
--tserver_master_addrs=127.22.252.190:35675
--never_fsync=true
--heap_profile_path=/tmp/kudu.30064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.130
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:15.520392 30064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:15.521451 30064 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:15.529695 30072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:15.530524 30069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:15.531359 30070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:15.532461 30064 server_base.cc:1034] running on GCE node
I20240501 14:09:15.532793 30064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:15.533315 30064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:15.534456 30064 hybrid_clock.cc:648] HybridClock initialized: now 1714572555534440 us; error 29 us; skew 500 ppm
I20240501 14:09:15.536110 30064 webserver.cc:460] Webserver started at http://127.22.252.130:35321/ using document root <none> and password file <none>
I20240501 14:09:15.536664 30064 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:15.536724 30064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:15.536885 30064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:15.538440 30064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d35c16dcb506426993811bb2519d7db3"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-bkzb"
I20240501 14:09:15.538883 30064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d35c16dcb506426993811bb2519d7db3"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-bkzb"
I20240501 14:09:15.541960 30064 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:09:15.543947 30078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.544848 30064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:15.544945 30064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d35c16dcb506426993811bb2519d7db3"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-bkzb"
I20240501 14:09:15.545023 30064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:15.568629 30064 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:15.569159 30064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:15.569301 30064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:15.569809 30064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:15.570611 30064 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:15.570657 30064 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.570714 30064 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:15.570741 30064 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.579056 30064 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.130:36457
I20240501 14:09:15.579097 30191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.130:36457 every 8 connection(s)
I20240501 14:09:15.579878 30064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:09:15.586059 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 30064
I20240501 14:09:15.586176 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:09:15.587951 30192 heartbeater.cc:347] Connected to a master server at 127.22.252.190:35675
I20240501 14:09:15.588141 30192 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:15.588353 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.131:0
--local_ip_for_outbound_sockets=127.22.252.131
--webserver_interface=127.22.252.131
--webserver_port=0
--tserver_master_addrs=127.22.252.190:35675
--builtin_ntp_servers=127.22.252.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:09:15.588587 30192 heartbeater.cc:511] Master 127.22.252.190:35675 requested a full tablet report, sending...
I20240501 14:09:15.589515 29873 ts_manager.cc:194] Registered new tserver with Master: d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:15.590261 29873 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.130:60833
W20240501 14:09:15.703673 30195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:15.703861 30195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:15.703927 30195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:15.706882 30195 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:15.706944 30195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:15.707052 30195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.131
I20240501 14:09:15.710572 30195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.252.131
--webserver_port=0
--tserver_master_addrs=127.22.252.190:35675
--never_fsync=true
--heap_profile_path=/tmp/kudu.30195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.131
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:15.711395 30195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:15.712496 30195 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:15.720501 30201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:15.721266 30200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:15.722182 30203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:15.723310 30195 server_base.cc:1034] running on GCE node
I20240501 14:09:15.723623 30195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:15.724143 30195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:15.725296 30195 hybrid_clock.cc:648] HybridClock initialized: now 1714572555725274 us; error 43 us; skew 500 ppm
I20240501 14:09:15.727442 30195 webserver.cc:460] Webserver started at http://127.22.252.131:45485/ using document root <none> and password file <none>
I20240501 14:09:15.727972 30195 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:15.728032 30195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:15.728220 30195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:15.730108 30195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "1d0bb69611af4876a2b3b33c4766e13a"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-bkzb"
I20240501 14:09:15.730677 30195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "1d0bb69611af4876a2b3b33c4766e13a"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-bkzb"
I20240501 14:09:15.736507 30195 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20240501 14:09:15.738862 30209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.739841 30195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:15.739941 30195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1d0bb69611af4876a2b3b33c4766e13a"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-bkzb"
I20240501 14:09:15.740026 30195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:15.774839 30195 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:15.775513 30195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:15.775664 30195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:15.776206 30195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:15.777081 30195 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:15.777127 30195 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.777191 30195 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:15.777216 30195 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.786724 30195 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.131:43103
I20240501 14:09:15.786764 30322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.131:43103 every 8 connection(s)
I20240501 14:09:15.787604 30195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:09:15.795643 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 30195
I20240501 14:09:15.795768 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:09:15.798319 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.132:0
--local_ip_for_outbound_sockets=127.22.252.132
--webserver_interface=127.22.252.132
--webserver_port=0
--tserver_master_addrs=127.22.252.190:35675
--builtin_ntp_servers=127.22.252.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:09:15.798447 30323 heartbeater.cc:347] Connected to a master server at 127.22.252.190:35675
I20240501 14:09:15.798651 30323 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:15.799044 30323 heartbeater.cc:511] Master 127.22.252.190:35675 requested a full tablet report, sending...
I20240501 14:09:15.799902 29873 ts_manager.cc:194] Registered new tserver with Master: 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:15.800462 29873 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.131:43891
W20240501 14:09:15.905947 30326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:15.906133 30326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:15.906193 30326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:15.909150 30326 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:15.909205 30326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:15.909302 30326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.132
I20240501 14:09:15.912724 30326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.252.132
--webserver_port=0
--tserver_master_addrs=127.22.252.190:35675
--never_fsync=true
--heap_profile_path=/tmp/kudu.30326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.132
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:15.913509 30326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:15.914595 30326 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:15.922891 30331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:15.923741 30334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:15.924510 30332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:15.925288 30326 server_base.cc:1034] running on GCE node
I20240501 14:09:15.925683 30326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:15.926172 30326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:15.927323 30326 hybrid_clock.cc:648] HybridClock initialized: now 1714572555927312 us; error 23 us; skew 500 ppm
I20240501 14:09:15.929088 30326 webserver.cc:460] Webserver started at http://127.22.252.132:45941/ using document root <none> and password file <none>
I20240501 14:09:15.929571 30326 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:15.929622 30326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:15.929769 30326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:15.931257 30326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-bkzb"
I20240501 14:09:15.931674 30326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-bkzb"
I20240501 14:09:15.934646 30326 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240501 14:09:15.936784 30340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.937736 30326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 14:09:15.937841 30326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-bkzb"
I20240501 14:09:15.937924 30326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:15.958189 30326 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:15.958761 30326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:15.958906 30326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:15.959393 30326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:15.960178 30326 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:15.960227 30326 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.960316 30326 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:15.960343 30326 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.968365 30326 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.132:34357
I20240501 14:09:15.968395 30453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.132:34357 every 8 connection(s)
I20240501 14:09:15.969166 30326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:09:15.975405 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 30326
I20240501 14:09:15.975533 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:09:15.976982 30454 heartbeater.cc:347] Connected to a master server at 127.22.252.190:35675
I20240501 14:09:15.977187 30454 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:15.977506 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.133:0
--local_ip_for_outbound_sockets=127.22.252.133
--webserver_interface=127.22.252.133
--webserver_port=0
--tserver_master_addrs=127.22.252.190:35675
--builtin_ntp_servers=127.22.252.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:09:15.977583 30454 heartbeater.cc:511] Master 127.22.252.190:35675 requested a full tablet report, sending...
I20240501 14:09:15.978428 29873 ts_manager.cc:194] Registered new tserver with Master: d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:15.978935 29873 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.132:49915
W20240501 14:09:16.082396 30457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:16.082580 30457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:16.082618 30457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:16.085458 30457 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:16.085512 30457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:16.085569 30457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.133
I20240501 14:09:16.088860 30457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.252.133
--webserver_port=0
--tserver_master_addrs=127.22.252.190:35675
--never_fsync=true
--heap_profile_path=/tmp/kudu.30457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.133
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:16.089637 30457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:16.090653 30457 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:16.098794 30463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:16.099545 30462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:16.100365 30465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:16.101545 30457 server_base.cc:1034] running on GCE node
I20240501 14:09:16.101893 30457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:16.102442 30457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:16.103583 30457 hybrid_clock.cc:648] HybridClock initialized: now 1714572556103558 us; error 36 us; skew 500 ppm
I20240501 14:09:16.105345 30457 webserver.cc:460] Webserver started at http://127.22.252.133:37943/ using document root <none> and password file <none>
I20240501 14:09:16.105839 30457 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:16.105893 30457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:16.106072 30457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:16.107604 30457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "a91029b90bf9415a89a2f70736759ad3"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-bkzb"
I20240501 14:09:16.108084 30457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "a91029b90bf9415a89a2f70736759ad3"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-bkzb"
I20240501 14:09:16.111284 30457 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240501 14:09:16.113452 30471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.114385 30457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:16.114486 30457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a91029b90bf9415a89a2f70736759ad3"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-bkzb"
I20240501 14:09:16.114569 30457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:16.136871 30457 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:16.137514 30457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:16.137670 30457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:16.138202 30457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:16.139119 30457 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:16.139168 30457 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.139230 30457 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:16.139258 30457 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.147434 30457 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.133:46265
I20240501 14:09:16.147460 30584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.133:46265 every 8 connection(s)
I20240501 14:09:16.148396 30457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:09:16.152823 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 30457
I20240501 14:09:16.152948 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:09:16.155226 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.134:0
--local_ip_for_outbound_sockets=127.22.252.134
--webserver_interface=127.22.252.134
--webserver_port=0
--tserver_master_addrs=127.22.252.190:35675
--builtin_ntp_servers=127.22.252.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:09:16.156978 30585 heartbeater.cc:347] Connected to a master server at 127.22.252.190:35675
I20240501 14:09:16.157217 30585 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:16.157641 30585 heartbeater.cc:511] Master 127.22.252.190:35675 requested a full tablet report, sending...
I20240501 14:09:16.158469 29873 ts_manager.cc:194] Registered new tserver with Master: a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:16.159226 29873 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.133:54157
W20240501 14:09:16.257972 30588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:16.258172 30588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:16.258213 30588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:16.261174 30588 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:16.261226 30588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:16.261317 30588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.134
I20240501 14:09:16.264724 30588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.252.134
--webserver_port=0
--tserver_master_addrs=127.22.252.190:35675
--never_fsync=true
--heap_profile_path=/tmp/kudu.30588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.134
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:16.265467 30588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:16.266500 30588 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:16.274840 30594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:16.275549 30593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:16.276324 30596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:16.277420 30588 server_base.cc:1034] running on GCE node
I20240501 14:09:16.277735 30588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:16.278208 30588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:16.279363 30588 hybrid_clock.cc:648] HybridClock initialized: now 1714572556279354 us; error 23 us; skew 500 ppm
I20240501 14:09:16.281126 30588 webserver.cc:460] Webserver started at http://127.22.252.134:38537/ using document root <none> and password file <none>
I20240501 14:09:16.281631 30588 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:16.281684 30588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:16.281829 30588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:16.283335 30588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-bkzb"
I20240501 14:09:16.283736 30588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-bkzb"
I20240501 14:09:16.286783 30588 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240501 14:09:16.288725 30602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.289623 30588 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:16.289718 30588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-bkzb"
I20240501 14:09:16.289793 30588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:16.306124 30588 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:16.306725 30588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:16.306874 30588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:16.307384 30588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:16.308257 30588 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:16.308310 30588 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.308374 30588 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:16.308399 30588 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.316445 30588 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.134:33497
I20240501 14:09:16.316483 30715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.134:33497 every 8 connection(s)
I20240501 14:09:16.317317 30588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:09:16.321559 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 30588
I20240501 14:09:16.321704 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:09:16.324024 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.135:0
--local_ip_for_outbound_sockets=127.22.252.135
--webserver_interface=127.22.252.135
--webserver_port=0
--tserver_master_addrs=127.22.252.190:35675
--builtin_ntp_servers=127.22.252.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:09:16.326304 30716 heartbeater.cc:347] Connected to a master server at 127.22.252.190:35675
I20240501 14:09:16.326503 30716 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:16.326918 30716 heartbeater.cc:511] Master 127.22.252.190:35675 requested a full tablet report, sending...
I20240501 14:09:16.327869 29873 ts_manager.cc:194] Registered new tserver with Master: 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:16.328456 29873 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.134:40205
I20240501 14:09:16.421612 30061 heartbeater.cc:503] Master 127.22.252.190:35675 was elected leader, sending a full tablet report...
W20240501 14:09:16.427182 30719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:16.427388 30719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:16.427459 30719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:16.430332 30719 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:16.430394 30719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:16.430485 30719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.135
I20240501 14:09:16.433773 30719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.252.135
--webserver_port=0
--tserver_master_addrs=127.22.252.190:35675
--never_fsync=true
--heap_profile_path=/tmp/kudu.30719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.135
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:16.434599 30719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:16.435638 30719 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:16.444051 30724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:16.444734 30725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:16.445772 30727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:16.446617 30719 server_base.cc:1034] running on GCE node
I20240501 14:09:16.446925 30719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:16.447420 30719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:16.448565 30719 hybrid_clock.cc:648] HybridClock initialized: now 1714572556448547 us; error 37 us; skew 500 ppm
I20240501 14:09:16.450390 30719 webserver.cc:460] Webserver started at http://127.22.252.135:44261/ using document root <none> and password file <none>
I20240501 14:09:16.450894 30719 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:16.450958 30719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:16.451123 30719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:16.452616 30719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "db935501e9104de5b15c8780681d6c38"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-bkzb"
I20240501 14:09:16.453054 30719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "db935501e9104de5b15c8780681d6c38"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-bkzb"
I20240501 14:09:16.455951 30719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:16.457906 30733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.458806 30719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:16.458904 30719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "db935501e9104de5b15c8780681d6c38"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-bkzb"
I20240501 14:09:16.458992 30719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:16.485124 30719 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:16.485646 30719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:16.485781 30719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:16.486250 30719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:16.487095 30719 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:16.487141 30719 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.487196 30719 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:16.487226 30719 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.495172 30719 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.135:41915
I20240501 14:09:16.495195 30846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.135:41915 every 8 connection(s)
I20240501 14:09:16.496032 30719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:09:16.499544 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 30719
I20240501 14:09:16.499670 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 14:09:16.501793 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.136:0
--local_ip_for_outbound_sockets=127.22.252.136
--webserver_interface=127.22.252.136
--webserver_port=0
--tserver_master_addrs=127.22.252.190:35675
--builtin_ntp_servers=127.22.252.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:09:16.504954 30847 heartbeater.cc:347] Connected to a master server at 127.22.252.190:35675
I20240501 14:09:16.505147 30847 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:16.505569 30847 heartbeater.cc:511] Master 127.22.252.190:35675 requested a full tablet report, sending...
I20240501 14:09:16.506505 29873 ts_manager.cc:194] Registered new tserver with Master: db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:16.507258 29873 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.135:47549
I20240501 14:09:16.592043 30192 heartbeater.cc:503] Master 127.22.252.190:35675 was elected leader, sending a full tablet report...
W20240501 14:09:16.604571 30850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:16.604763 30850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:16.604835 30850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:16.607753 30850 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:16.607816 30850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:16.607908 30850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.136
I20240501 14:09:16.611335 30850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.252.136
--webserver_port=0
--tserver_master_addrs=127.22.252.190:35675
--never_fsync=true
--heap_profile_path=/tmp/kudu.30850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.136
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:16.612099 30850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:16.613119 30850 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:16.621373 30855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:16.622102 30858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:16.623157 30856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:16.624015 30850 server_base.cc:1034] running on GCE node
I20240501 14:09:16.624377 30850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:16.624881 30850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:16.626039 30850 hybrid_clock.cc:648] HybridClock initialized: now 1714572556626028 us; error 29 us; skew 500 ppm
I20240501 14:09:16.627820 30850 webserver.cc:460] Webserver started at http://127.22.252.136:41581/ using document root <none> and password file <none>
I20240501 14:09:16.628382 30850 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:16.628440 30850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:16.628616 30850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:16.630152 30850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-bkzb"
I20240501 14:09:16.630594 30850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-bkzb"
I20240501 14:09:16.633733 30850 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240501 14:09:16.635730 30864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.636672 30850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:16.636782 30850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-bkzb"
I20240501 14:09:16.636859 30850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:16.657158 30850 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:16.657702 30850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:16.657837 30850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:16.658363 30850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:16.659180 30850 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:16.659227 30850 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.659281 30850 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:16.659312 30850 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.667431 30850 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.136:45937
I20240501 14:09:16.667472 30977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.136:45937 every 8 connection(s)
I20240501 14:09:16.668321 30850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:09:16.676107 30978 heartbeater.cc:347] Connected to a master server at 127.22.252.190:35675
I20240501 14:09:16.676330 30978 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:16.676728 30978 heartbeater.cc:511] Master 127.22.252.190:35675 requested a full tablet report, sending...
I20240501 14:09:16.677546 29873 ts_manager.cc:194] Registered new tserver with Master: 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:16.677665 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 30850
I20240501 14:09:16.677784 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 14:09:16.678083 29873 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.136:41453
I20240501 14:09:16.679860 23538 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:09:16.688076 23538 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:09:16.688134 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 30457
I20240501 14:09:16.693717 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 30588
I20240501 14:09:16.698877 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 30719
I20240501 14:09:16.704094 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 30850
I20240501 14:09:16.802181 30323 heartbeater.cc:503] Master 127.22.252.190:35675 was elected leader, sending a full tablet report...
I20240501 14:09:16.980677 30454 heartbeater.cc:503] Master 127.22.252.190:35675 was elected leader, sending a full tablet report...
I20240501 14:09:20.460467 29873 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:20.480106 30257 tablet_service.cc:1467] Processing CreateTablet for tablet 27944e6f68e742b78ef5b350650aaee2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4435db0f5a574d0285fbd28671f7b39d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:20.480248 30126 tablet_service.cc:1467] Processing CreateTablet for tablet 27944e6f68e742b78ef5b350650aaee2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4435db0f5a574d0285fbd28671f7b39d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:20.480089 29993 tablet_service.cc:1467] Processing CreateTablet for tablet e22ffa3b919b4b10995ba3a73157b221 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4435db0f5a574d0285fbd28671f7b39d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:20.480314 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 985c116f66ec4a6cb10e3ec1d5f6366f (DEFAULT_TABLE table=rebalance_test_table_0 [id=4435db0f5a574d0285fbd28671f7b39d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:20.481051 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27944e6f68e742b78ef5b350650aaee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:20.481051 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27944e6f68e742b78ef5b350650aaee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:20.481051 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e22ffa3b919b4b10995ba3a73157b221. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:20.481051 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 985c116f66ec4a6cb10e3ec1d5f6366f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:20.481490 29995 tablet_service.cc:1467] Processing CreateTablet for tablet 27944e6f68e742b78ef5b350650aaee2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4435db0f5a574d0285fbd28671f7b39d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:20.481626 29994 tablet_service.cc:1467] Processing CreateTablet for tablet 985c116f66ec4a6cb10e3ec1d5f6366f (DEFAULT_TABLE table=rebalance_test_table_0 [id=4435db0f5a574d0285fbd28671f7b39d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:20.481740 29995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27944e6f68e742b78ef5b350650aaee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:20.481846 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 985c116f66ec4a6cb10e3ec1d5f6366f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:20.484484 30387 tablet_service.cc:1467] Processing CreateTablet for tablet e22ffa3b919b4b10995ba3a73157b221 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4435db0f5a574d0285fbd28671f7b39d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:20.484481 30125 tablet_service.cc:1467] Processing CreateTablet for tablet 985c116f66ec4a6cb10e3ec1d5f6366f (DEFAULT_TABLE table=rebalance_test_table_0 [id=4435db0f5a574d0285fbd28671f7b39d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:20.484814 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e22ffa3b919b4b10995ba3a73157b221. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:20.485523 30256 tablet_service.cc:1467] Processing CreateTablet for tablet e22ffa3b919b4b10995ba3a73157b221 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4435db0f5a574d0285fbd28671f7b39d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:20.485759 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e22ffa3b919b4b10995ba3a73157b221. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:20.486434 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 985c116f66ec4a6cb10e3ec1d5f6366f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:20.490159 31000 tablet_bootstrap.cc:492] T 985c116f66ec4a6cb10e3ec1d5f6366f P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:20.491338 31001 tablet_bootstrap.cc:492] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:20.492075 30998 tablet_bootstrap.cc:492] T 27944e6f68e742b78ef5b350650aaee2 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:20.492267 31000 tablet_bootstrap.cc:654] T 985c116f66ec4a6cb10e3ec1d5f6366f P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:20.492483 30999 tablet_bootstrap.cc:492] T e22ffa3b919b4b10995ba3a73157b221 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:20.493095 31000 log.cc:824] T 985c116f66ec4a6cb10e3ec1d5f6366f P d35c16dcb506426993811bb2519d7db3: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:20.493184 31001 tablet_bootstrap.cc:654] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:20.493896 30998 tablet_bootstrap.cc:654] T 27944e6f68e742b78ef5b350650aaee2 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:20.493952 31001 log.cc:824] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:20.494262 30999 tablet_bootstrap.cc:654] T e22ffa3b919b4b10995ba3a73157b221 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:20.494866 31000 tablet_bootstrap.cc:492] T 985c116f66ec4a6cb10e3ec1d5f6366f P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:20.495046 31000 ts_tablet_manager.cc:1367] T 985c116f66ec4a6cb10e3ec1d5f6366f P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:09:20.495054 30998 log.cc:824] T 27944e6f68e742b78ef5b350650aaee2 P 93f2518fbadf4f9991cda93afacb363e: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:20.495434 30999 log.cc:824] T e22ffa3b919b4b10995ba3a73157b221 P d1c217469ae4478ebd49e4ed9f7352d7: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:20.495566 31001 tablet_bootstrap.cc:492] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:20.495708 31001 ts_tablet_manager.cc:1367] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:09:20.496887 30998 tablet_bootstrap.cc:492] T 27944e6f68e742b78ef5b350650aaee2 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:20.497031 30998 ts_tablet_manager.cc:1367] T 27944e6f68e742b78ef5b350650aaee2 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:09:20.497151 30999 tablet_bootstrap.cc:492] T e22ffa3b919b4b10995ba3a73157b221 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:20.497292 30999 ts_tablet_manager.cc:1367] T e22ffa3b919b4b10995ba3a73157b221 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:09:20.499073 31000 raft_consensus.cc:348] T 985c116f66ec4a6cb10e3ec1d5f6366f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:20.499373 31000 raft_consensus.cc:374] T 985c116f66ec4a6cb10e3ec1d5f6366f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:20.499467 31000 raft_consensus.cc:729] T 985c116f66ec4a6cb10e3ec1d5f6366f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:20.499508 31001 raft_consensus.cc:348] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:20.499689 31001 raft_consensus.cc:374] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:20.499743 31001 raft_consensus.cc:729] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:20.500124 31000 consensus_queue.cc:260] T 985c116f66ec4a6cb10e3ec1d5f6366f P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:20.500151 31001 consensus_queue.cc:260] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:20.500329 30998 raft_consensus.cc:348] T 27944e6f68e742b78ef5b350650aaee2 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:20.500520 30998 raft_consensus.cc:374] T 27944e6f68e742b78ef5b350650aaee2 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:20.500583 30998 raft_consensus.cc:729] T 27944e6f68e742b78ef5b350650aaee2 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:20.501062 31001 ts_tablet_manager.cc:1398] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 14:09:20.501272 31000 ts_tablet_manager.cc:1398] T 985c116f66ec4a6cb10e3ec1d5f6366f P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20240501 14:09:20.501272 30998 consensus_queue.cc:260] T 27944e6f68e742b78ef5b350650aaee2 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:20.501453 31001 tablet_bootstrap.cc:492] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:20.501724 31000 tablet_bootstrap.cc:492] T 27944e6f68e742b78ef5b350650aaee2 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:20.502082 30998 ts_tablet_manager.cc:1398] T 27944e6f68e742b78ef5b350650aaee2 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 14:09:20.502400 30998 tablet_bootstrap.cc:492] T e22ffa3b919b4b10995ba3a73157b221 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:20.502820 31000 tablet_bootstrap.cc:654] T 27944e6f68e742b78ef5b350650aaee2 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:20.503266 30998 tablet_bootstrap.cc:654] T e22ffa3b919b4b10995ba3a73157b221 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:20.503417 31001 tablet_bootstrap.cc:654] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:20.504832 31001 tablet_bootstrap.cc:492] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:20.504909 31001 ts_tablet_manager.cc:1367] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:20.505352 30998 tablet_bootstrap.cc:492] T e22ffa3b919b4b10995ba3a73157b221 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:20.505437 30998 ts_tablet_manager.cc:1367] T e22ffa3b919b4b10995ba3a73157b221 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:20.505470 31001 raft_consensus.cc:348] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:20.505563 31001 raft_consensus.cc:374] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:20.505599 31001 raft_consensus.cc:729] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:20.505795 31001 consensus_queue.cc:260] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:20.505883 31000 tablet_bootstrap.cc:492] T 27944e6f68e742b78ef5b350650aaee2 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:20.505967 31000 ts_tablet_manager.cc:1367] T 27944e6f68e742b78ef5b350650aaee2 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:09:20.505945 30998 raft_consensus.cc:348] T e22ffa3b919b4b10995ba3a73157b221 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:20.506033 30998 raft_consensus.cc:374] T e22ffa3b919b4b10995ba3a73157b221 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:20.506064 30998 raft_consensus.cc:729] T e22ffa3b919b4b10995ba3a73157b221 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:20.506080 31001 ts_tablet_manager.cc:1398] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:20.506166 30998 consensus_queue.cc:260] T e22ffa3b919b4b10995ba3a73157b221 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:20.506420 30998 ts_tablet_manager.cc:1398] T e22ffa3b919b4b10995ba3a73157b221 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:20.506464 31000 raft_consensus.cc:348] T 27944e6f68e742b78ef5b350650aaee2 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:20.506558 31000 raft_consensus.cc:374] T 27944e6f68e742b78ef5b350650aaee2 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:20.506564 30998 tablet_bootstrap.cc:492] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:20.506589 31000 raft_consensus.cc:729] T 27944e6f68e742b78ef5b350650aaee2 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:20.506724 31000 consensus_queue.cc:260] T 27944e6f68e742b78ef5b350650aaee2 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:20.506999 31000 ts_tablet_manager.cc:1398] T 27944e6f68e742b78ef5b350650aaee2 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:20.507388 30998 tablet_bootstrap.cc:654] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:20.508535 30998 tablet_bootstrap.cc:492] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:20.508615 30998 ts_tablet_manager.cc:1367] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:20.508548 30999 raft_consensus.cc:348] T e22ffa3b919b4b10995ba3a73157b221 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:20.508742 30999 raft_consensus.cc:374] T e22ffa3b919b4b10995ba3a73157b221 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:20.508795 30999 raft_consensus.cc:729] T e22ffa3b919b4b10995ba3a73157b221 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:20.509168 30999 consensus_queue.cc:260] T e22ffa3b919b4b10995ba3a73157b221 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:20.509234 30998 raft_consensus.cc:348] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:20.509321 30998 raft_consensus.cc:374] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:20.509362 30998 raft_consensus.cc:729] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:20.509482 30998 consensus_queue.cc:260] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:20.509768 30998 ts_tablet_manager.cc:1398] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:20.509961 30999 ts_tablet_manager.cc:1398] T e22ffa3b919b4b10995ba3a73157b221 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20240501 14:09:20.510464 30999 tablet_bootstrap.cc:492] T 985c116f66ec4a6cb10e3ec1d5f6366f P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:20.511456 30999 tablet_bootstrap.cc:654] T 985c116f66ec4a6cb10e3ec1d5f6366f P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:20.512522 30999 tablet_bootstrap.cc:492] T 985c116f66ec4a6cb10e3ec1d5f6366f P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:20.512600 30999 ts_tablet_manager.cc:1367] T 985c116f66ec4a6cb10e3ec1d5f6366f P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:20.513056 30999 raft_consensus.cc:348] T 985c116f66ec4a6cb10e3ec1d5f6366f P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:20.513135 30999 raft_consensus.cc:374] T 985c116f66ec4a6cb10e3ec1d5f6366f P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:20.513168 30999 raft_consensus.cc:729] T 985c116f66ec4a6cb10e3ec1d5f6366f P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:20.513295 30999 consensus_queue.cc:260] T 985c116f66ec4a6cb10e3ec1d5f6366f P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:20.513531 30999 ts_tablet_manager.cc:1398] T 985c116f66ec4a6cb10e3ec1d5f6366f P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:20.521802 31008 raft_consensus.cc:482] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:20.521950 31008 raft_consensus.cc:504] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:20.522861 31008 leader_election.cc:288] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:20.525300 31006 raft_consensus.cc:482] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:20.525439 31006 raft_consensus.cc:504] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:20.526415 31006 leader_election.cc:288] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:20.526753 30408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985c116f66ec4a6cb10e3ec1d5f6366f" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:20.526994 30408 raft_consensus.cc:2454] T 985c116f66ec4a6cb10e3ec1d5f6366f P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:20.527278 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985c116f66ec4a6cb10e3ec1d5f6366f" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:20.527369 29948 leader_election.cc:302] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:20.527513 30146 raft_consensus.cc:2454] T 985c116f66ec4a6cb10e3ec1d5f6366f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:20.527624 31008 raft_consensus.cc:2781] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:20.527688 31008 raft_consensus.cc:482] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:20.527760 31008 raft_consensus.cc:3037] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:20.528836 31008 raft_consensus.cc:504] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:20.529208 31008 leader_election.cc:288] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:20.529458 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985c116f66ec4a6cb10e3ec1d5f6366f" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:20.529556 30146 raft_consensus.cc:3037] T 985c116f66ec4a6cb10e3ec1d5f6366f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:20.529819 30408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985c116f66ec4a6cb10e3ec1d5f6366f" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:20.529909 30408 raft_consensus.cc:3037] T 985c116f66ec4a6cb10e3ec1d5f6366f P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:20.530589 30146 raft_consensus.cc:2454] T 985c116f66ec4a6cb10e3ec1d5f6366f P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:20.530880 29951 leader_election.cc:302] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:20.530915 30408 raft_consensus.cc:2454] T 985c116f66ec4a6cb10e3ec1d5f6366f P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:20.531034 31008 raft_consensus.cc:2781] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:20.531257 31008 raft_consensus.cc:686] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:20.531394 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27944e6f68e742b78ef5b350650aaee2" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:20.531522 30146 raft_consensus.cc:2454] T 27944e6f68e742b78ef5b350650aaee2 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:20.531826 30213 leader_election.cc:302] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:20.531867 30015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27944e6f68e742b78ef5b350650aaee2" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:20.532011 30015 raft_consensus.cc:2454] T 27944e6f68e742b78ef5b350650aaee2 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:20.531889 31008 consensus_queue.cc:237] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:20.532028 31006 raft_consensus.cc:2781] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:20.532092 31006 raft_consensus.cc:482] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:20.532141 31006 raft_consensus.cc:3037] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:20.533121 31006 raft_consensus.cc:504] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:20.533445 31006 leader_election.cc:288] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:20.533754 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27944e6f68e742b78ef5b350650aaee2" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:20.533859 30146 raft_consensus.cc:3037] T 27944e6f68e742b78ef5b350650aaee2 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:20.534114 30015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27944e6f68e742b78ef5b350650aaee2" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:20.534210 30015 raft_consensus.cc:3037] T 27944e6f68e742b78ef5b350650aaee2 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:20.534709 30146 raft_consensus.cc:2454] T 27944e6f68e742b78ef5b350650aaee2 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:20.534472 29871 catalog_manager.cc:5484] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:20.534986 30015 raft_consensus.cc:2454] T 27944e6f68e742b78ef5b350650aaee2 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:20.534999 30213 leader_election.cc:302] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:20.535146 31006 raft_consensus.cc:2781] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:20.535357 31006 raft_consensus.cc:686] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:20.535635 31006 consensus_queue.cc:237] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:20.538117 29871 catalog_manager.cc:5484] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
W20240501 14:09:20.541944 30324 tablet.cc:2369] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:09:20.584635 30193 tablet.cc:2369] T 27944e6f68e742b78ef5b350650aaee2 P d35c16dcb506426993811bb2519d7db3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:09:20.662770 30062 tablet.cc:2369] T 27944e6f68e742b78ef5b350650aaee2 P 93f2518fbadf4f9991cda93afacb363e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:09:20.723631 30455 tablet.cc:2369] T 985c116f66ec4a6cb10e3ec1d5f6366f P d1c217469ae4478ebd49e4ed9f7352d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:20.797191 31006 raft_consensus.cc:482] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:20.797303 31006 raft_consensus.cc:504] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:20.797747 31006 leader_election.cc:288] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:20.798184 30015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e22ffa3b919b4b10995ba3a73157b221" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:20.798408 30015 raft_consensus.cc:2454] T e22ffa3b919b4b10995ba3a73157b221 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:20.798705 30212 leader_election.cc:302] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:20.798858 31006 raft_consensus.cc:2781] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:20.798908 31006 raft_consensus.cc:482] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:20.798961 31006 raft_consensus.cc:3037] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:20.799818 31006 raft_consensus.cc:504] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:20.800108 31006 leader_election.cc:288] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:20.800485 30015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e22ffa3b919b4b10995ba3a73157b221" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:20.800606 30015 raft_consensus.cc:3037] T e22ffa3b919b4b10995ba3a73157b221 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:20.801559 30015 raft_consensus.cc:2454] T e22ffa3b919b4b10995ba3a73157b221 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:20.801570 30408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e22ffa3b919b4b10995ba3a73157b221" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:20.801628 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e22ffa3b919b4b10995ba3a73157b221" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:20.801702 30408 raft_consensus.cc:2454] T e22ffa3b919b4b10995ba3a73157b221 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:20.801780 30212 leader_election.cc:302] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:20.801944 31006 raft_consensus.cc:2781] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:20.802018 31006 raft_consensus.cc:686] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:20.802197 31006 consensus_queue.cc:237] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:20.803907 29871 catalog_manager.cc:5484] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
W20240501 14:09:20.940615 31024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:20.940879 31024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:20.943862 31024 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 14:09:21.022887 30015 raft_consensus.cc:1261] T 27944e6f68e742b78ef5b350650aaee2 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:21.023851 30407 raft_consensus.cc:1261] T 985c116f66ec4a6cb10e3ec1d5f6366f P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:21.023903 30146 raft_consensus.cc:1261] T 27944e6f68e742b78ef5b350650aaee2 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:21.024461 31008 consensus_queue.cc:1035] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:21.025220 30408 raft_consensus.cc:3037] T e22ffa3b919b4b10995ba3a73157b221 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.026247 30408 raft_consensus.cc:1261] T e22ffa3b919b4b10995ba3a73157b221 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:21.023907 30145 raft_consensus.cc:1261] T 985c116f66ec4a6cb10e3ec1d5f6366f P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:21.027048 31006 consensus_queue.cc:1035] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:21.027297 30015 raft_consensus.cc:1261] T e22ffa3b919b4b10995ba3a73157b221 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:21.027397 31059 consensus_queue.cc:1035] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:21.027511 31006 consensus_queue.cc:1035] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:21.027966 31006 consensus_queue.cc:1035] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:21.028023 31008 consensus_queue.cc:1035] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:21.036695 31054 mvcc.cc:205] Tried to move back new op lower bound from 7022889209963646976 to 7022889207941615616. Current Snapshot: MvccSnapshot[applied={T|T < 7022889209946296320}]
I20240501 14:09:21.040339 31056 mvcc.cc:205] Tried to move back new op lower bound from 7022889209963786240 to 7022889207956217856. Current Snapshot: MvccSnapshot[applied={T|T < 7022889209943207936}]
I20240501 14:09:21.051988 31071 mvcc.cc:205] Tried to move back new op lower bound from 7022889209963786240 to 7022889207956217856. Current Snapshot: MvccSnapshot[applied={T|T < 7022889209943207936}]
I20240501 14:09:21.061484 31058 mvcc.cc:205] Tried to move back new op lower bound from 7022889209946296320 to 7022889207941615616. Current Snapshot: MvccSnapshot[applied={T|T < 7022889209946296320}]
INSERT report
rows total: 2000
time total: 103.866 ms
time per row: 0.0519328 ms
I20240501 14:09:21.091333 29871 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:21.096261 30387 tablet_service.cc:1467] Processing CreateTablet for tablet f740d894942a4bb584ca5688ef823d55 (DEFAULT_TABLE table=rebalance_test_table_1 [id=df055757a36d4b9f89886608b024f0b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.096218 30126 tablet_service.cc:1467] Processing CreateTablet for tablet f740d894942a4bb584ca5688ef823d55 (DEFAULT_TABLE table=rebalance_test_table_1 [id=df055757a36d4b9f89886608b024f0b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.096536 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f740d894942a4bb584ca5688ef823d55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.096536 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f740d894942a4bb584ca5688ef823d55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.097316 30257 tablet_service.cc:1467] Processing CreateTablet for tablet f740d894942a4bb584ca5688ef823d55 (DEFAULT_TABLE table=rebalance_test_table_1 [id=df055757a36d4b9f89886608b024f0b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.097594 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f740d894942a4bb584ca5688ef823d55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.098084 30125 tablet_service.cc:1467] Processing CreateTablet for tablet cfc701c19cd74d0db2f90ce580230177 (DEFAULT_TABLE table=rebalance_test_table_1 [id=df055757a36d4b9f89886608b024f0b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.098345 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc701c19cd74d0db2f90ce580230177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.098965 30256 tablet_service.cc:1467] Processing CreateTablet for tablet cfc701c19cd74d0db2f90ce580230177 (DEFAULT_TABLE table=rebalance_test_table_1 [id=df055757a36d4b9f89886608b024f0b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.099028 29994 tablet_service.cc:1467] Processing CreateTablet for tablet fb36e198bcf34ef68a9595d7cbe93e80 (DEFAULT_TABLE table=rebalance_test_table_1 [id=df055757a36d4b9f89886608b024f0b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.099189 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc701c19cd74d0db2f90ce580230177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.099269 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb36e198bcf34ef68a9595d7cbe93e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.100095 30124 tablet_service.cc:1467] Processing CreateTablet for tablet fb36e198bcf34ef68a9595d7cbe93e80 (DEFAULT_TABLE table=rebalance_test_table_1 [id=df055757a36d4b9f89886608b024f0b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.100343 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb36e198bcf34ef68a9595d7cbe93e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.101156 30255 tablet_service.cc:1467] Processing CreateTablet for tablet fb36e198bcf34ef68a9595d7cbe93e80 (DEFAULT_TABLE table=rebalance_test_table_1 [id=df055757a36d4b9f89886608b024f0b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.101400 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb36e198bcf34ef68a9595d7cbe93e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.101667 30388 tablet_service.cc:1467] Processing CreateTablet for tablet cfc701c19cd74d0db2f90ce580230177 (DEFAULT_TABLE table=rebalance_test_table_1 [id=df055757a36d4b9f89886608b024f0b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.101894 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc701c19cd74d0db2f90ce580230177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.101941 31093 tablet_bootstrap.cc:492] T f740d894942a4bb584ca5688ef823d55 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:21.102507 31094 tablet_bootstrap.cc:492] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:21.103535 31094 tablet_bootstrap.cc:654] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.103535 31093 tablet_bootstrap.cc:654] T f740d894942a4bb584ca5688ef823d55 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.103792 31092 tablet_bootstrap.cc:492] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:21.104728 31092 tablet_bootstrap.cc:654] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.104794 31093 tablet_bootstrap.cc:492] T f740d894942a4bb584ca5688ef823d55 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:21.104794 31094 tablet_bootstrap.cc:492] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:21.104873 31093 ts_tablet_manager.cc:1367] T f740d894942a4bb584ca5688ef823d55 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240501 14:09:21.104873 31094 ts_tablet_manager.cc:1367] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:21.105346 31091 tablet_bootstrap.cc:492] T f740d894942a4bb584ca5688ef823d55 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:21.105296 31093 raft_consensus.cc:348] T f740d894942a4bb584ca5688ef823d55 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:21.105297 31094 raft_consensus.cc:348] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:21.105396 31093 raft_consensus.cc:374] T f740d894942a4bb584ca5688ef823d55 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.105396 31094 raft_consensus.cc:374] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.105430 31093 raft_consensus.cc:729] T f740d894942a4bb584ca5688ef823d55 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.105430 31094 raft_consensus.cc:729] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.105556 31094 consensus_queue.cc:260] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:21.105556 31093 consensus_queue.cc:260] T f740d894942a4bb584ca5688ef823d55 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:21.105820 31093 ts_tablet_manager.cc:1398] T f740d894942a4bb584ca5688ef823d55 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:21.105820 31094 ts_tablet_manager.cc:1398] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:21.105958 31093 tablet_bootstrap.cc:492] T cfc701c19cd74d0db2f90ce580230177 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:21.106417 31092 tablet_bootstrap.cc:492] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:21.106489 31092 ts_tablet_manager.cc:1367] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:21.106803 31093 tablet_bootstrap.cc:654] T cfc701c19cd74d0db2f90ce580230177 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.106937 31092 raft_consensus.cc:348] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:21.107519 31091 tablet_bootstrap.cc:654] T f740d894942a4bb584ca5688ef823d55 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.107615 31092 raft_consensus.cc:374] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.107815 31093 tablet_bootstrap.cc:492] T cfc701c19cd74d0db2f90ce580230177 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:21.107905 31093 ts_tablet_manager.cc:1367] T cfc701c19cd74d0db2f90ce580230177 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:21.108250 31092 raft_consensus.cc:729] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.108446 31092 consensus_queue.cc:260] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:21.108526 31093 raft_consensus.cc:348] T cfc701c19cd74d0db2f90ce580230177 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:21.108636 31093 raft_consensus.cc:374] T cfc701c19cd74d0db2f90ce580230177 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.108670 31093 raft_consensus.cc:729] T cfc701c19cd74d0db2f90ce580230177 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.108789 31093 consensus_queue.cc:260] T cfc701c19cd74d0db2f90ce580230177 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:21.108906 31092 ts_tablet_manager.cc:1398] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:21.109061 31093 ts_tablet_manager.cc:1398] T cfc701c19cd74d0db2f90ce580230177 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:21.109217 31093 tablet_bootstrap.cc:492] T fb36e198bcf34ef68a9595d7cbe93e80 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:21.109365 31091 tablet_bootstrap.cc:492] T f740d894942a4bb584ca5688ef823d55 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:21.109566 31092 tablet_bootstrap.cc:492] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:21.110085 31093 tablet_bootstrap.cc:654] T fb36e198bcf34ef68a9595d7cbe93e80 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.110628 31091 ts_tablet_manager.cc:1367] T f740d894942a4bb584ca5688ef823d55 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:21.111097 31091 raft_consensus.cc:348] T f740d894942a4bb584ca5688ef823d55 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:21.111265 31091 raft_consensus.cc:374] T f740d894942a4bb584ca5688ef823d55 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.110498 31092 tablet_bootstrap.cc:654] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.111351 31091 raft_consensus.cc:729] T f740d894942a4bb584ca5688ef823d55 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.111496 31091 consensus_queue.cc:260] T f740d894942a4bb584ca5688ef823d55 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:21.111881 31091 ts_tablet_manager.cc:1398] T f740d894942a4bb584ca5688ef823d55 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:21.112107 31091 tablet_bootstrap.cc:492] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:21.112594 31093 tablet_bootstrap.cc:492] T fb36e198bcf34ef68a9595d7cbe93e80 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:21.112671 31093 ts_tablet_manager.cc:1367] T fb36e198bcf34ef68a9595d7cbe93e80 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:21.112994 31093 raft_consensus.cc:348] T fb36e198bcf34ef68a9595d7cbe93e80 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:21.113067 31091 tablet_bootstrap.cc:654] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.113088 31093 raft_consensus.cc:374] T fb36e198bcf34ef68a9595d7cbe93e80 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.113127 31093 raft_consensus.cc:729] T fb36e198bcf34ef68a9595d7cbe93e80 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.113305 31093 consensus_queue.cc:260] T fb36e198bcf34ef68a9595d7cbe93e80 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:21.113631 31093 ts_tablet_manager.cc:1398] T fb36e198bcf34ef68a9595d7cbe93e80 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:21.113940 31092 tablet_bootstrap.cc:492] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:21.114027 31092 ts_tablet_manager.cc:1367] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:09:21.114068 31091 tablet_bootstrap.cc:492] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:21.114145 31091 ts_tablet_manager.cc:1367] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:21.114542 31092 raft_consensus.cc:348] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:21.114603 31091 raft_consensus.cc:348] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:21.114648 31092 raft_consensus.cc:374] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.114688 31092 raft_consensus.cc:729] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.114691 31091 raft_consensus.cc:374] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.114728 31091 raft_consensus.cc:729] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.114818 31092 consensus_queue.cc:260] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:21.114845 31091 consensus_queue.cc:260] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:21.115083 31091 ts_tablet_manager.cc:1398] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:21.115126 31092 ts_tablet_manager.cc:1398] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:21.115272 31092 tablet_bootstrap.cc:492] T fb36e198bcf34ef68a9595d7cbe93e80 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:21.116348 31092 tablet_bootstrap.cc:654] T fb36e198bcf34ef68a9595d7cbe93e80 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.117645 31092 tablet_bootstrap.cc:492] T fb36e198bcf34ef68a9595d7cbe93e80 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:21.117722 31092 ts_tablet_manager.cc:1367] T fb36e198bcf34ef68a9595d7cbe93e80 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:09:21.118115 31092 raft_consensus.cc:348] T fb36e198bcf34ef68a9595d7cbe93e80 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:21.118201 31092 raft_consensus.cc:374] T fb36e198bcf34ef68a9595d7cbe93e80 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.118223 31092 raft_consensus.cc:729] T fb36e198bcf34ef68a9595d7cbe93e80 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.118356 31092 consensus_queue.cc:260] T fb36e198bcf34ef68a9595d7cbe93e80 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:21.118558 31092 ts_tablet_manager.cc:1398] T fb36e198bcf34ef68a9595d7cbe93e80 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:21.214906 31069 raft_consensus.cc:482] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:21.215057 31069 raft_consensus.cc:504] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:21.216109 31069 leader_election.cc:288] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:21.220111 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc701c19cd74d0db2f90ce580230177" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:21.220170 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc701c19cd74d0db2f90ce580230177" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:21.220333 30145 raft_consensus.cc:2454] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:21.220342 30264 raft_consensus.cc:2454] T cfc701c19cd74d0db2f90ce580230177 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:21.220690 30344 leader_election.cc:302] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:21.220912 31069 raft_consensus.cc:2781] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:21.220971 31069 raft_consensus.cc:482] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:21.221010 31069 raft_consensus.cc:3037] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.221812 31069 raft_consensus.cc:504] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:21.222095 31069 leader_election.cc:288] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:21.222409 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc701c19cd74d0db2f90ce580230177" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:21.222425 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc701c19cd74d0db2f90ce580230177" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:21.222498 30264 raft_consensus.cc:3037] T cfc701c19cd74d0db2f90ce580230177 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.222513 30145 raft_consensus.cc:3037] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.223224 30264 raft_consensus.cc:2454] T cfc701c19cd74d0db2f90ce580230177 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:21.223250 30145 raft_consensus.cc:2454] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:21.223484 30343 leader_election.cc:302] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:21.223697 31069 raft_consensus.cc:2781] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:21.223798 31069 raft_consensus.cc:686] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:21.224076 31069 consensus_queue.cc:237] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:21.226155 29872 catalog_manager.cc:5484] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:21.279165 31061 raft_consensus.cc:482] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:21.279326 31061 raft_consensus.cc:504] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:21.280422 31061 leader_election.cc:288] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:21.284804 30408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f740d894942a4bb584ca5688ef823d55" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:21.284974 30408 raft_consensus.cc:2454] T f740d894942a4bb584ca5688ef823d55 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:21.285239 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f740d894942a4bb584ca5688ef823d55" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:21.285291 30079 leader_election.cc:302] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:21.285375 30264 raft_consensus.cc:2454] T f740d894942a4bb584ca5688ef823d55 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:21.285498 31061 raft_consensus.cc:2781] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:21.285560 31061 raft_consensus.cc:482] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:21.285598 31061 raft_consensus.cc:3037] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.286451 31061 raft_consensus.cc:504] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:21.286715 31061 leader_election.cc:288] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:21.287030 30408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f740d894942a4bb584ca5688ef823d55" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:21.287060 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f740d894942a4bb584ca5688ef823d55" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:21.287143 30408 raft_consensus.cc:3037] T f740d894942a4bb584ca5688ef823d55 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.287146 30264 raft_consensus.cc:3037] T f740d894942a4bb584ca5688ef823d55 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.288108 30408 raft_consensus.cc:2454] T f740d894942a4bb584ca5688ef823d55 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:21.288381 30079 leader_election.cc:302] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:21.288487 30264 raft_consensus.cc:2454] T f740d894942a4bb584ca5688ef823d55 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:21.288559 31061 raft_consensus.cc:2781] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:21.288640 31061 raft_consensus.cc:686] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:21.288905 31061 consensus_queue.cc:237] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:21.291052 29872 catalog_manager.cc:5484] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:21.455932 31065 raft_consensus.cc:482] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:21.456068 31065 raft_consensus.cc:504] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:21.456568 31065 leader_election.cc:288] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:21.457103 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb36e198bcf34ef68a9595d7cbe93e80" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:21.457270 30145 raft_consensus.cc:2454] T fb36e198bcf34ef68a9595d7cbe93e80 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:21.457624 29951 leader_election.cc:302] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:21.457816 31065 raft_consensus.cc:2781] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:21.457875 31065 raft_consensus.cc:482] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:21.457917 31065 raft_consensus.cc:3037] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.458804 31065 raft_consensus.cc:504] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:21.459144 31065 leader_election.cc:288] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:21.459452 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb36e198bcf34ef68a9595d7cbe93e80" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:21.459551 30145 raft_consensus.cc:3037] T fb36e198bcf34ef68a9595d7cbe93e80 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.460413 30145 raft_consensus.cc:2454] T fb36e198bcf34ef68a9595d7cbe93e80 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:21.460638 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb36e198bcf34ef68a9595d7cbe93e80" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:21.460666 29951 leader_election.cc:302] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:21.460696 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb36e198bcf34ef68a9595d7cbe93e80" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:21.460752 30277 raft_consensus.cc:3037] T fb36e198bcf34ef68a9595d7cbe93e80 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.460848 31065 raft_consensus.cc:2781] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:21.460975 31065 raft_consensus.cc:686] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:21.461120 31065 consensus_queue.cc:237] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:21.461813 30277 raft_consensus.cc:2454] T fb36e198bcf34ef68a9595d7cbe93e80 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:21.462777 29872 catalog_manager.cc:5484] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
W20240501 14:09:21.654186 31113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:21.654394 31113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:21.657316 31113 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:09:21.664898 31069 consensus_queue.cc:1035] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:21.668354 31069 consensus_queue.cc:1035] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240501 14:09:21.748558 30145 raft_consensus.cc:1261] T fb36e198bcf34ef68a9595d7cbe93e80 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:21.748979 30264 raft_consensus.cc:1261] T fb36e198bcf34ef68a9595d7cbe93e80 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:21.749095 31065 consensus_queue.cc:1035] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:21.749660 31065 consensus_queue.cc:1035] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:21.750864 30276 raft_consensus.cc:1261] T f740d894942a4bb584ca5688ef823d55 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:21.751302 31061 consensus_queue.cc:1035] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:21.751523 30408 raft_consensus.cc:1261] T f740d894942a4bb584ca5688ef823d55 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:21.752434 31080 consensus_queue.cc:1035] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 108.41 ms
time per row: 0.0542052 ms
I20240501 14:09:21.807935 29872 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:21.813256 30124 tablet_service.cc:1467] Processing CreateTablet for tablet 048651987f8d4311b8b6bfb73b60ee90 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2f015d5186014d24b820e1f454091285]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.813254 29994 tablet_service.cc:1467] Processing CreateTablet for tablet 048651987f8d4311b8b6bfb73b60ee90 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2f015d5186014d24b820e1f454091285]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.813556 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 048651987f8d4311b8b6bfb73b60ee90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.813556 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 048651987f8d4311b8b6bfb73b60ee90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.813910 30386 tablet_service.cc:1467] Processing CreateTablet for tablet d67171f19dcc4dc3bbe937b35b0f9456 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2f015d5186014d24b820e1f454091285]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.813874 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 048651987f8d4311b8b6bfb73b60ee90 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2f015d5186014d24b820e1f454091285]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.814155 30386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67171f19dcc4dc3bbe937b35b0f9456. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.814303 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 048651987f8d4311b8b6bfb73b60ee90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.814165 30387 tablet_service.cc:1467] Processing CreateTablet for tablet df5c7fba18cc4cccb8c61b0895c91340 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2f015d5186014d24b820e1f454091285]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.815351 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df5c7fba18cc4cccb8c61b0895c91340. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.816118 29993 tablet_service.cc:1467] Processing CreateTablet for tablet d67171f19dcc4dc3bbe937b35b0f9456 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2f015d5186014d24b820e1f454091285]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.816169 30255 tablet_service.cc:1467] Processing CreateTablet for tablet df5c7fba18cc4cccb8c61b0895c91340 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2f015d5186014d24b820e1f454091285]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.816378 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67171f19dcc4dc3bbe937b35b0f9456. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.816421 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df5c7fba18cc4cccb8c61b0895c91340. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.816756 29994 tablet_service.cc:1467] Processing CreateTablet for tablet df5c7fba18cc4cccb8c61b0895c91340 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2f015d5186014d24b820e1f454091285]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.816974 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df5c7fba18cc4cccb8c61b0895c91340. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.818432 31176 tablet_bootstrap.cc:492] T df5c7fba18cc4cccb8c61b0895c91340 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:21.818868 31174 tablet_bootstrap.cc:492] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:21.819384 31176 tablet_bootstrap.cc:654] T df5c7fba18cc4cccb8c61b0895c91340 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.819414 30124 tablet_service.cc:1467] Processing CreateTablet for tablet d67171f19dcc4dc3bbe937b35b0f9456 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2f015d5186014d24b820e1f454091285]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.819586 31174 tablet_bootstrap.cc:654] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.819650 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67171f19dcc4dc3bbe937b35b0f9456. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.820147 31173 tablet_bootstrap.cc:492] T 048651987f8d4311b8b6bfb73b60ee90 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:21.820510 31174 tablet_bootstrap.cc:492] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:21.820613 31176 tablet_bootstrap.cc:492] T df5c7fba18cc4cccb8c61b0895c91340 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:21.820654 31174 ts_tablet_manager.cc:1367] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:21.820721 31176 ts_tablet_manager.cc:1367] T df5c7fba18cc4cccb8c61b0895c91340 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:21.821084 31173 tablet_bootstrap.cc:654] T 048651987f8d4311b8b6bfb73b60ee90 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.821190 31176 raft_consensus.cc:348] T df5c7fba18cc4cccb8c61b0895c91340 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:21.821321 31176 raft_consensus.cc:374] T df5c7fba18cc4cccb8c61b0895c91340 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.821357 31176 raft_consensus.cc:729] T df5c7fba18cc4cccb8c61b0895c91340 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.821486 31176 consensus_queue.cc:260] T df5c7fba18cc4cccb8c61b0895c91340 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:21.821763 31175 tablet_bootstrap.cc:492] T d67171f19dcc4dc3bbe937b35b0f9456 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:21.821756 31176 ts_tablet_manager.cc:1398] T df5c7fba18cc4cccb8c61b0895c91340 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:21.822366 31173 tablet_bootstrap.cc:492] T 048651987f8d4311b8b6bfb73b60ee90 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:21.822479 31173 ts_tablet_manager.cc:1367] T 048651987f8d4311b8b6bfb73b60ee90 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:21.822682 31175 tablet_bootstrap.cc:654] T d67171f19dcc4dc3bbe937b35b0f9456 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.822993 31173 raft_consensus.cc:348] T 048651987f8d4311b8b6bfb73b60ee90 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:21.823130 31173 raft_consensus.cc:374] T 048651987f8d4311b8b6bfb73b60ee90 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.823246 31173 raft_consensus.cc:729] T 048651987f8d4311b8b6bfb73b60ee90 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.823283 31174 raft_consensus.cc:348] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:21.823390 31174 raft_consensus.cc:374] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.823431 31174 raft_consensus.cc:729] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.823434 31173 consensus_queue.cc:260] T 048651987f8d4311b8b6bfb73b60ee90 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:21.823551 31174 consensus_queue.cc:260] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:21.823778 31173 ts_tablet_manager.cc:1398] T 048651987f8d4311b8b6bfb73b60ee90 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:21.823819 31174 ts_tablet_manager.cc:1398] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:21.823964 31174 tablet_bootstrap.cc:492] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:21.823989 31173 tablet_bootstrap.cc:492] T d67171f19dcc4dc3bbe937b35b0f9456 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:21.824719 31174 tablet_bootstrap.cc:654] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.825101 31173 tablet_bootstrap.cc:654] T d67171f19dcc4dc3bbe937b35b0f9456 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.825785 31175 tablet_bootstrap.cc:492] T d67171f19dcc4dc3bbe937b35b0f9456 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:21.825865 31175 ts_tablet_manager.cc:1367] T d67171f19dcc4dc3bbe937b35b0f9456 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:21.826215 31173 tablet_bootstrap.cc:492] T d67171f19dcc4dc3bbe937b35b0f9456 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:21.826294 31173 ts_tablet_manager.cc:1367] T d67171f19dcc4dc3bbe937b35b0f9456 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:21.826324 31175 raft_consensus.cc:348] T d67171f19dcc4dc3bbe937b35b0f9456 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:21.826450 31175 raft_consensus.cc:374] T d67171f19dcc4dc3bbe937b35b0f9456 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.826488 31175 raft_consensus.cc:729] T d67171f19dcc4dc3bbe937b35b0f9456 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.826607 31175 consensus_queue.cc:260] T d67171f19dcc4dc3bbe937b35b0f9456 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:21.826736 31173 raft_consensus.cc:348] T d67171f19dcc4dc3bbe937b35b0f9456 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:21.826820 31173 raft_consensus.cc:374] T d67171f19dcc4dc3bbe937b35b0f9456 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.826849 31173 raft_consensus.cc:729] T d67171f19dcc4dc3bbe937b35b0f9456 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.826869 31175 ts_tablet_manager.cc:1398] T d67171f19dcc4dc3bbe937b35b0f9456 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:21.826972 31173 consensus_queue.cc:260] T d67171f19dcc4dc3bbe937b35b0f9456 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:21.827030 31175 tablet_bootstrap.cc:492] T 048651987f8d4311b8b6bfb73b60ee90 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:21.827243 31173 ts_tablet_manager.cc:1398] T d67171f19dcc4dc3bbe937b35b0f9456 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:21.827842 31175 tablet_bootstrap.cc:654] T 048651987f8d4311b8b6bfb73b60ee90 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.828500 31174 tablet_bootstrap.cc:492] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:21.828583 31174 ts_tablet_manager.cc:1367] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:21.829018 31174 raft_consensus.cc:348] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:21.829108 31174 raft_consensus.cc:374] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.829138 31174 raft_consensus.cc:729] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.829272 31174 consensus_queue.cc:260] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:21.829566 31174 ts_tablet_manager.cc:1398] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:21.829675 31175 tablet_bootstrap.cc:492] T 048651987f8d4311b8b6bfb73b60ee90 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:21.829720 31174 tablet_bootstrap.cc:492] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:21.829744 31175 ts_tablet_manager.cc:1367] T 048651987f8d4311b8b6bfb73b60ee90 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:21.830338 31175 raft_consensus.cc:348] T 048651987f8d4311b8b6bfb73b60ee90 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:21.830556 31174 tablet_bootstrap.cc:654] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.830672 31175 raft_consensus.cc:374] T 048651987f8d4311b8b6bfb73b60ee90 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.830767 31175 raft_consensus.cc:729] T 048651987f8d4311b8b6bfb73b60ee90 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.830951 31175 consensus_queue.cc:260] T 048651987f8d4311b8b6bfb73b60ee90 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:21.831501 31175 ts_tablet_manager.cc:1398] T 048651987f8d4311b8b6bfb73b60ee90 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:21.831568 31174 tablet_bootstrap.cc:492] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:21.831640 31174 ts_tablet_manager.cc:1367] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:09:21.831661 31175 tablet_bootstrap.cc:492] T df5c7fba18cc4cccb8c61b0895c91340 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:21.832031 31174 raft_consensus.cc:348] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:21.832120 31174 raft_consensus.cc:374] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.832158 31174 raft_consensus.cc:729] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.832284 31174 consensus_queue.cc:260] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:21.832530 31174 ts_tablet_manager.cc:1398] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:21.833161 31175 tablet_bootstrap.cc:654] T df5c7fba18cc4cccb8c61b0895c91340 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.834254 31175 tablet_bootstrap.cc:492] T df5c7fba18cc4cccb8c61b0895c91340 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:21.834342 31175 ts_tablet_manager.cc:1367] T df5c7fba18cc4cccb8c61b0895c91340 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:21.834652 31175 raft_consensus.cc:348] T df5c7fba18cc4cccb8c61b0895c91340 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:21.834731 31175 raft_consensus.cc:374] T df5c7fba18cc4cccb8c61b0895c91340 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.834765 31175 raft_consensus.cc:729] T df5c7fba18cc4cccb8c61b0895c91340 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.834909 31175 consensus_queue.cc:260] T df5c7fba18cc4cccb8c61b0895c91340 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:21.835172 31175 ts_tablet_manager.cc:1398] T df5c7fba18cc4cccb8c61b0895c91340 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:21.836540 31065 raft_consensus.cc:482] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:21.836624 31065 raft_consensus.cc:504] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:21.836978 31065 leader_election.cc:288] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:21.837291 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "048651987f8d4311b8b6bfb73b60ee90" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:21.837412 30146 raft_consensus.cc:2454] T 048651987f8d4311b8b6bfb73b60ee90 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:21.837586 30408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "048651987f8d4311b8b6bfb73b60ee90" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:21.837697 30408 raft_consensus.cc:2454] T 048651987f8d4311b8b6bfb73b60ee90 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:21.837697 29951 leader_election.cc:302] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:21.838171 31065 raft_consensus.cc:2781] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:21.838236 31065 raft_consensus.cc:482] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:21.838302 31065 raft_consensus.cc:3037] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.838999 31065 raft_consensus.cc:504] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:21.839270 31065 leader_election.cc:288] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:21.839546 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "048651987f8d4311b8b6bfb73b60ee90" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:21.839573 30408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "048651987f8d4311b8b6bfb73b60ee90" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:21.839645 30408 raft_consensus.cc:3037] T 048651987f8d4311b8b6bfb73b60ee90 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.839646 30146 raft_consensus.cc:3037] T 048651987f8d4311b8b6bfb73b60ee90 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.840384 30408 raft_consensus.cc:2454] T 048651987f8d4311b8b6bfb73b60ee90 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:21.840399 30146 raft_consensus.cc:2454] T 048651987f8d4311b8b6bfb73b60ee90 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:21.840615 29948 leader_election.cc:302] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:21.840787 31065 raft_consensus.cc:2781] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:21.840883 31065 raft_consensus.cc:686] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:21.841027 31065 consensus_queue.cc:237] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:21.842348 29873 catalog_manager.cc:5484] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:21.849735 31065 raft_consensus.cc:482] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:21.849821 31065 raft_consensus.cc:504] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:21.850142 31065 leader_election.cc:288] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:21.850420 30408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67171f19dcc4dc3bbe937b35b0f9456" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:21.850430 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67171f19dcc4dc3bbe937b35b0f9456" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:21.850512 30408 raft_consensus.cc:2454] T d67171f19dcc4dc3bbe937b35b0f9456 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:21.850538 30146 raft_consensus.cc:2454] T d67171f19dcc4dc3bbe937b35b0f9456 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:21.850744 29948 leader_election.cc:302] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:21.850899 31065 raft_consensus.cc:2781] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:21.850957 31065 raft_consensus.cc:482] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:21.850989 31065 raft_consensus.cc:3037] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.851605 31065 raft_consensus.cc:504] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:21.851866 31065 leader_election.cc:288] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:21.852222 30408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67171f19dcc4dc3bbe937b35b0f9456" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:21.852222 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67171f19dcc4dc3bbe937b35b0f9456" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:21.852353 30408 raft_consensus.cc:3037] T d67171f19dcc4dc3bbe937b35b0f9456 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.852353 30146 raft_consensus.cc:3037] T d67171f19dcc4dc3bbe937b35b0f9456 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.853206 30146 raft_consensus.cc:2454] T d67171f19dcc4dc3bbe937b35b0f9456 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:21.853206 30408 raft_consensus.cc:2454] T d67171f19dcc4dc3bbe937b35b0f9456 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:21.853466 29951 leader_election.cc:302] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:21.853626 31065 raft_consensus.cc:2781] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:21.853708 31065 raft_consensus.cc:686] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:21.853829 31065 consensus_queue.cc:237] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:21.855314 29873 catalog_manager.cc:5484] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:21.871716 31065 raft_consensus.cc:482] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:21.871804 31065 raft_consensus.cc:504] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:21.872087 31065 leader_election.cc:288] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:21.872463 30408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df5c7fba18cc4cccb8c61b0895c91340" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:21.872449 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df5c7fba18cc4cccb8c61b0895c91340" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:21.872576 30408 raft_consensus.cc:2454] T df5c7fba18cc4cccb8c61b0895c91340 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:21.872581 30277 raft_consensus.cc:2454] T df5c7fba18cc4cccb8c61b0895c91340 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:21.872824 29950 leader_election.cc:302] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:21.872970 31065 raft_consensus.cc:2781] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:21.873018 31065 raft_consensus.cc:482] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:21.873051 31065 raft_consensus.cc:3037] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.873688 31065 raft_consensus.cc:504] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:21.873970 31065 leader_election.cc:288] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:21.874233 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df5c7fba18cc4cccb8c61b0895c91340" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:21.874296 30408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df5c7fba18cc4cccb8c61b0895c91340" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:21.874337 30277 raft_consensus.cc:3037] T df5c7fba18cc4cccb8c61b0895c91340 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.874385 30408 raft_consensus.cc:3037] T df5c7fba18cc4cccb8c61b0895c91340 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.875218 30408 raft_consensus.cc:2454] T df5c7fba18cc4cccb8c61b0895c91340 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:21.875218 30277 raft_consensus.cc:2454] T df5c7fba18cc4cccb8c61b0895c91340 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:21.875474 29950 leader_election.cc:302] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:21.875633 31065 raft_consensus.cc:2781] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:21.875705 31065 raft_consensus.cc:686] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:21.875845 31065 consensus_queue.cc:237] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:21.877271 29873 catalog_manager.cc:5484] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
W20240501 14:09:21.991124 31189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:21.991494 31189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:21.995704 31189 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 14:09:22.087584 30408 raft_consensus.cc:1261] T 048651987f8d4311b8b6bfb73b60ee90 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.087769 30146 raft_consensus.cc:1261] T 048651987f8d4311b8b6bfb73b60ee90 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.088269 31019 consensus_queue.cc:1035] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:22.088627 31019 consensus_queue.cc:1035] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:22.089326 30277 raft_consensus.cc:1261] T df5c7fba18cc4cccb8c61b0895c91340 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.090298 30406 raft_consensus.cc:1261] T d67171f19dcc4dc3bbe937b35b0f9456 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.091295 31065 consensus_queue.cc:1035] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:22.091761 30408 raft_consensus.cc:1261] T df5c7fba18cc4cccb8c61b0895c91340 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.093793 31065 consensus_queue.cc:1035] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:22.094064 31065 consensus_queue.cc:1035] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:22.095536 30145 raft_consensus.cc:1261] T d67171f19dcc4dc3bbe937b35b0f9456 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.099517 31065 consensus_queue.cc:1035] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 117.673 ms
time per row: 0.0588366 ms
I20240501 14:09:22.153752 29873 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:22.158730 30255 tablet_service.cc:1467] Processing CreateTablet for tablet 0517ff3519644b489b912f8e5a36274c (DEFAULT_TABLE table=rebalance_test_table_3 [id=b7b3f5bc4b6649c5a97ec9f02da4e967]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.158771 29994 tablet_service.cc:1467] Processing CreateTablet for tablet 0517ff3519644b489b912f8e5a36274c (DEFAULT_TABLE table=rebalance_test_table_3 [id=b7b3f5bc4b6649c5a97ec9f02da4e967]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.159034 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0517ff3519644b489b912f8e5a36274c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.159052 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0517ff3519644b489b912f8e5a36274c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.159720 30125 tablet_service.cc:1467] Processing CreateTablet for tablet 53c51f3845c94f47b043621d98f70229 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b7b3f5bc4b6649c5a97ec9f02da4e967]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.159866 30124 tablet_service.cc:1467] Processing CreateTablet for tablet 0517ff3519644b489b912f8e5a36274c (DEFAULT_TABLE table=rebalance_test_table_3 [id=b7b3f5bc4b6649c5a97ec9f02da4e967]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.159999 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53c51f3845c94f47b043621d98f70229. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.160120 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0517ff3519644b489b912f8e5a36274c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.161264 31176 tablet_bootstrap.cc:492] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:22.162061 31176 tablet_bootstrap.cc:654] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.162509 31173 tablet_bootstrap.cc:492] T 0517ff3519644b489b912f8e5a36274c P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:22.163098 31176 tablet_bootstrap.cc:492] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:22.163166 31176 ts_tablet_manager.cc:1367] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:22.163333 31173 tablet_bootstrap.cc:654] T 0517ff3519644b489b912f8e5a36274c P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.163623 31176 raft_consensus.cc:348] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.163720 31176 raft_consensus.cc:374] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.163749 31176 raft_consensus.cc:729] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.163880 31176 consensus_queue.cc:260] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.163962 31174 tablet_bootstrap.cc:492] T 0517ff3519644b489b912f8e5a36274c P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:22.164144 31176 ts_tablet_manager.cc:1398] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:22.164635 31174 tablet_bootstrap.cc:654] T 0517ff3519644b489b912f8e5a36274c P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.165980 30387 tablet_service.cc:1467] Processing CreateTablet for tablet af39c304693645269a6af59190e95cad (DEFAULT_TABLE table=rebalance_test_table_3 [id=b7b3f5bc4b6649c5a97ec9f02da4e967]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.166226 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af39c304693645269a6af59190e95cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.167996 31175 tablet_bootstrap.cc:492] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:22.168159 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 53c51f3845c94f47b043621d98f70229 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b7b3f5bc4b6649c5a97ec9f02da4e967]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.168361 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53c51f3845c94f47b043621d98f70229. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.168846 31175 tablet_bootstrap.cc:654] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.168826 30256 tablet_service.cc:1467] Processing CreateTablet for tablet 53c51f3845c94f47b043621d98f70229 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b7b3f5bc4b6649c5a97ec9f02da4e967]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.169062 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53c51f3845c94f47b043621d98f70229. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.169018 29994 tablet_service.cc:1467] Processing CreateTablet for tablet af39c304693645269a6af59190e95cad (DEFAULT_TABLE table=rebalance_test_table_3 [id=b7b3f5bc4b6649c5a97ec9f02da4e967]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.169226 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af39c304693645269a6af59190e95cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.170738 31175 tablet_bootstrap.cc:492] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:22.170821 31175 ts_tablet_manager.cc:1367] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:22.171088 31173 tablet_bootstrap.cc:492] T 0517ff3519644b489b912f8e5a36274c P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:22.171154 31173 ts_tablet_manager.cc:1367] T 0517ff3519644b489b912f8e5a36274c P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:22.171288 31175 raft_consensus.cc:348] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:22.171388 31175 raft_consensus.cc:374] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.171432 31175 raft_consensus.cc:729] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.171537 31175 consensus_queue.cc:260] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:22.171600 31173 raft_consensus.cc:348] T 0517ff3519644b489b912f8e5a36274c P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.171710 31176 tablet_bootstrap.cc:492] T 53c51f3845c94f47b043621d98f70229 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:22.171720 31173 raft_consensus.cc:374] T 0517ff3519644b489b912f8e5a36274c P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.171752 31173 raft_consensus.cc:729] T 0517ff3519644b489b912f8e5a36274c P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.171818 31175 ts_tablet_manager.cc:1398] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:22.171857 31173 consensus_queue.cc:260] T 0517ff3519644b489b912f8e5a36274c P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.171962 31175 tablet_bootstrap.cc:492] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:22.172112 31173 ts_tablet_manager.cc:1398] T 0517ff3519644b489b912f8e5a36274c P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:22.172268 31173 tablet_bootstrap.cc:492] T 53c51f3845c94f47b043621d98f70229 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:22.172798 31175 tablet_bootstrap.cc:654] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.172899 31176 tablet_bootstrap.cc:654] T 53c51f3845c94f47b043621d98f70229 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.173132 31173 tablet_bootstrap.cc:654] T 53c51f3845c94f47b043621d98f70229 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.173561 30255 tablet_service.cc:1467] Processing CreateTablet for tablet af39c304693645269a6af59190e95cad (DEFAULT_TABLE table=rebalance_test_table_3 [id=b7b3f5bc4b6649c5a97ec9f02da4e967]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.173784 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af39c304693645269a6af59190e95cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.174561 31174 tablet_bootstrap.cc:492] T 0517ff3519644b489b912f8e5a36274c P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:22.174649 31174 ts_tablet_manager.cc:1367] T 0517ff3519644b489b912f8e5a36274c P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240501 14:09:22.175076 31174 raft_consensus.cc:348] T 0517ff3519644b489b912f8e5a36274c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.175173 31174 raft_consensus.cc:374] T 0517ff3519644b489b912f8e5a36274c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.175206 31174 raft_consensus.cc:729] T 0517ff3519644b489b912f8e5a36274c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.175323 31174 consensus_queue.cc:260] T 0517ff3519644b489b912f8e5a36274c P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.175575 31174 ts_tablet_manager.cc:1398] T 0517ff3519644b489b912f8e5a36274c P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:22.175849 31174 tablet_bootstrap.cc:492] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:22.176090 31176 tablet_bootstrap.cc:492] T 53c51f3845c94f47b043621d98f70229 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:22.176158 31176 ts_tablet_manager.cc:1367] T 53c51f3845c94f47b043621d98f70229 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:22.176620 31176 raft_consensus.cc:348] T 53c51f3845c94f47b043621d98f70229 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.176707 31176 raft_consensus.cc:374] T 53c51f3845c94f47b043621d98f70229 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.176703 31174 tablet_bootstrap.cc:654] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.176729 31176 raft_consensus.cc:729] T 53c51f3845c94f47b043621d98f70229 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.176823 31176 consensus_queue.cc:260] T 53c51f3845c94f47b043621d98f70229 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.177079 31176 ts_tablet_manager.cc:1398] T 53c51f3845c94f47b043621d98f70229 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:22.177266 31176 tablet_bootstrap.cc:492] T af39c304693645269a6af59190e95cad P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:22.178478 31176 tablet_bootstrap.cc:654] T af39c304693645269a6af59190e95cad P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.179214 31174 tablet_bootstrap.cc:492] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:22.179294 31174 ts_tablet_manager.cc:1367] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:22.179687 31174 raft_consensus.cc:348] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:22.179772 31174 raft_consensus.cc:374] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.179798 31174 raft_consensus.cc:729] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.179875 31176 tablet_bootstrap.cc:492] T af39c304693645269a6af59190e95cad P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:22.179903 31174 consensus_queue.cc:260] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:22.179955 31176 ts_tablet_manager.cc:1367] T af39c304693645269a6af59190e95cad P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:22.180127 31174 ts_tablet_manager.cc:1398] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:22.180608 31176 raft_consensus.cc:348] T af39c304693645269a6af59190e95cad P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:22.180712 31176 raft_consensus.cc:374] T af39c304693645269a6af59190e95cad P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.180732 31173 tablet_bootstrap.cc:492] T 53c51f3845c94f47b043621d98f70229 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:22.180749 31176 raft_consensus.cc:729] T af39c304693645269a6af59190e95cad P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.180800 31173 ts_tablet_manager.cc:1367] T 53c51f3845c94f47b043621d98f70229 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:22.180874 31176 consensus_queue.cc:260] T af39c304693645269a6af59190e95cad P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:22.181149 31176 ts_tablet_manager.cc:1398] T af39c304693645269a6af59190e95cad P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:22.181169 31173 raft_consensus.cc:348] T 53c51f3845c94f47b043621d98f70229 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.181255 31173 raft_consensus.cc:374] T 53c51f3845c94f47b043621d98f70229 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.181280 31173 raft_consensus.cc:729] T 53c51f3845c94f47b043621d98f70229 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.181382 31173 consensus_queue.cc:260] T 53c51f3845c94f47b043621d98f70229 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.181581 31173 ts_tablet_manager.cc:1398] T 53c51f3845c94f47b043621d98f70229 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:22.182670 31175 tablet_bootstrap.cc:492] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:22.182749 31175 ts_tablet_manager.cc:1367] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:09:22.183319 31175 raft_consensus.cc:348] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.183410 31175 raft_consensus.cc:374] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.183434 31175 raft_consensus.cc:729] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.183524 31175 consensus_queue.cc:260] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.183732 31175 ts_tablet_manager.cc:1398] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:22.277213 31215 raft_consensus.cc:482] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:22.277348 31215 raft_consensus.cc:504] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:22.277779 31215 leader_election.cc:288] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:22.278182 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af39c304693645269a6af59190e95cad" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:22.278194 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af39c304693645269a6af59190e95cad" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:22.278340 30407 raft_consensus.cc:2454] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:22.278350 30277 raft_consensus.cc:2454] T af39c304693645269a6af59190e95cad P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:22.278638 29948 leader_election.cc:302] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:22.278780 31215 raft_consensus.cc:2781] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:22.278837 31215 raft_consensus.cc:482] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:22.278883 31215 raft_consensus.cc:3037] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.279685 31215 raft_consensus.cc:504] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:22.280009 31215 leader_election.cc:288] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:22.280315 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af39c304693645269a6af59190e95cad" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:22.280436 30407 raft_consensus.cc:3037] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.281216 30407 raft_consensus.cc:2454] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:22.281446 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af39c304693645269a6af59190e95cad" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:22.281479 29948 leader_election.cc:302] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:22.281560 30277 raft_consensus.cc:3037] T af39c304693645269a6af59190e95cad P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.281659 31215 raft_consensus.cc:2781] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:22.281741 31215 raft_consensus.cc:686] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:22.281874 31215 consensus_queue.cc:237] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:22.282420 30277 raft_consensus.cc:2454] T af39c304693645269a6af59190e95cad P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:22.283664 29868 catalog_manager.cc:5484] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:22.300849 31114 raft_consensus.cc:482] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:22.300983 31114 raft_consensus.cc:504] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.301434 31114 leader_election.cc:288] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:22.301869 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0517ff3519644b489b912f8e5a36274c" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:22.301873 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0517ff3519644b489b912f8e5a36274c" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:22.302007 30146 raft_consensus.cc:2454] T 0517ff3519644b489b912f8e5a36274c P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:22.302009 30014 raft_consensus.cc:2454] T 0517ff3519644b489b912f8e5a36274c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:22.302310 30213 leader_election.cc:302] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:22.302520 31114 raft_consensus.cc:2781] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:22.302565 31114 raft_consensus.cc:482] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:22.302610 31114 raft_consensus.cc:3037] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.303337 31114 raft_consensus.cc:504] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.303624 31114 leader_election.cc:288] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:22.303936 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0517ff3519644b489b912f8e5a36274c" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:22.303993 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0517ff3519644b489b912f8e5a36274c" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:22.304069 30014 raft_consensus.cc:3037] T 0517ff3519644b489b912f8e5a36274c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.304090 30146 raft_consensus.cc:3037] T 0517ff3519644b489b912f8e5a36274c P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.305034 30014 raft_consensus.cc:2454] T 0517ff3519644b489b912f8e5a36274c P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:22.305034 30146 raft_consensus.cc:2454] T 0517ff3519644b489b912f8e5a36274c P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:22.305325 30212 leader_election.cc:302] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:22.305506 31114 raft_consensus.cc:2781] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:22.305647 31114 raft_consensus.cc:686] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:22.305809 31114 consensus_queue.cc:237] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.307360 29868 catalog_manager.cc:5484] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:22.426996 31076 raft_consensus.cc:482] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:22.427135 31076 raft_consensus.cc:504] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.427579 31076 leader_election.cc:288] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:22.428021 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53c51f3845c94f47b043621d98f70229" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:22.428022 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53c51f3845c94f47b043621d98f70229" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:22.428155 30146 raft_consensus.cc:2454] T 53c51f3845c94f47b043621d98f70229 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:22.428155 30277 raft_consensus.cc:2454] T 53c51f3845c94f47b043621d98f70229 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:22.428490 30343 leader_election.cc:302] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:22.428679 31076 raft_consensus.cc:2781] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:22.428725 31076 raft_consensus.cc:482] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:22.428757 31076 raft_consensus.cc:3037] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.429739 31076 raft_consensus.cc:504] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.430083 31076 leader_election.cc:288] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:22.430399 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53c51f3845c94f47b043621d98f70229" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:22.430410 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53c51f3845c94f47b043621d98f70229" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:22.430501 30277 raft_consensus.cc:3037] T 53c51f3845c94f47b043621d98f70229 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.430531 30146 raft_consensus.cc:3037] T 53c51f3845c94f47b043621d98f70229 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.431226 30277 raft_consensus.cc:2454] T 53c51f3845c94f47b043621d98f70229 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:22.431391 30146 raft_consensus.cc:2454] T 53c51f3845c94f47b043621d98f70229 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:22.431483 30343 leader_election.cc:302] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:22.431656 31076 raft_consensus.cc:2781] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:22.431813 31076 raft_consensus.cc:686] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:22.431973 31076 consensus_queue.cc:237] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.433584 29868 catalog_manager.cc:5484] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
W20240501 14:09:22.557521 31240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:22.557760 31240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:22.560878 31240 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 14:09:22.639181 30407 raft_consensus.cc:1261] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.639181 30277 raft_consensus.cc:1261] T af39c304693645269a6af59190e95cad P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.640758 31019 consensus_queue.cc:1035] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:22.641561 30277 raft_consensus.cc:1261] T 53c51f3845c94f47b043621d98f70229 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.641563 30145 raft_consensus.cc:1261] T 0517ff3519644b489b912f8e5a36274c P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.641983 31019 consensus_queue.cc:1035] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:22.642226 31236 consensus_queue.cc:1035] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:22.643745 30014 raft_consensus.cc:1261] T 0517ff3519644b489b912f8e5a36274c P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.644450 31076 consensus_queue.cc:1035] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:22.646714 30145 raft_consensus.cc:1261] T 53c51f3845c94f47b043621d98f70229 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.647082 31114 consensus_queue.cc:1035] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:22.653355 31076 consensus_queue.cc:1035] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 102.522 ms
time per row: 0.0512611 ms
I20240501 14:09:22.704224 29866 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:22.709978 30255 tablet_service.cc:1467] Processing CreateTablet for tablet ef6731d4b3824d3eaeba37e2fb464d32 (DEFAULT_TABLE table=rebalance_test_table_4 [id=979bbb7a065445c89d4cf2f3e28ed0d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.710022 29994 tablet_service.cc:1467] Processing CreateTablet for tablet ef6731d4b3824d3eaeba37e2fb464d32 (DEFAULT_TABLE table=rebalance_test_table_4 [id=979bbb7a065445c89d4cf2f3e28ed0d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.710302 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6731d4b3824d3eaeba37e2fb464d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.710305 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6731d4b3824d3eaeba37e2fb464d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.710853 30125 tablet_service.cc:1467] Processing CreateTablet for tablet ef6731d4b3824d3eaeba37e2fb464d32 (DEFAULT_TABLE table=rebalance_test_table_4 [id=979bbb7a065445c89d4cf2f3e28ed0d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.711133 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6731d4b3824d3eaeba37e2fb464d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.712585 31301 tablet_bootstrap.cc:492] T ef6731d4b3824d3eaeba37e2fb464d32 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:22.712853 31302 tablet_bootstrap.cc:492] T ef6731d4b3824d3eaeba37e2fb464d32 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:22.713064 30255 tablet_service.cc:1467] Processing CreateTablet for tablet 15f40a19911b40fb980eead07bcbd106 (DEFAULT_TABLE table=rebalance_test_table_4 [id=979bbb7a065445c89d4cf2f3e28ed0d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.713304 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15f40a19911b40fb980eead07bcbd106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.713251 30124 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3e51eb67b947038487dc0c7782d0b6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=979bbb7a065445c89d4cf2f3e28ed0d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.713459 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3e51eb67b947038487dc0c7782d0b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.713537 31301 tablet_bootstrap.cc:654] T ef6731d4b3824d3eaeba37e2fb464d32 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.713732 31302 tablet_bootstrap.cc:654] T ef6731d4b3824d3eaeba37e2fb464d32 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.714587 31302 tablet_bootstrap.cc:492] T ef6731d4b3824d3eaeba37e2fb464d32 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:22.714645 31302 ts_tablet_manager.cc:1367] T ef6731d4b3824d3eaeba37e2fb464d32 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:22.714955 31302 raft_consensus.cc:348] T ef6731d4b3824d3eaeba37e2fb464d32 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.715029 31302 raft_consensus.cc:374] T ef6731d4b3824d3eaeba37e2fb464d32 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.715049 31302 raft_consensus.cc:729] T ef6731d4b3824d3eaeba37e2fb464d32 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.715204 31302 consensus_queue.cc:260] T ef6731d4b3824d3eaeba37e2fb464d32 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.715487 31302 ts_tablet_manager.cc:1398] T ef6731d4b3824d3eaeba37e2fb464d32 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:22.715658 31302 tablet_bootstrap.cc:492] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:22.715695 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3e51eb67b947038487dc0c7782d0b6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=979bbb7a065445c89d4cf2f3e28ed0d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.715972 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3e51eb67b947038487dc0c7782d0b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.716280 31301 tablet_bootstrap.cc:492] T ef6731d4b3824d3eaeba37e2fb464d32 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:22.716354 31301 ts_tablet_manager.cc:1367] T ef6731d4b3824d3eaeba37e2fb464d32 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:22.716578 31302 tablet_bootstrap.cc:654] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.716856 31301 raft_consensus.cc:348] T ef6731d4b3824d3eaeba37e2fb464d32 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.716989 31301 raft_consensus.cc:374] T ef6731d4b3824d3eaeba37e2fb464d32 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.717023 31301 raft_consensus.cc:729] T ef6731d4b3824d3eaeba37e2fb464d32 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.717154 31301 consensus_queue.cc:260] T ef6731d4b3824d3eaeba37e2fb464d32 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.717454 31301 ts_tablet_manager.cc:1398] T ef6731d4b3824d3eaeba37e2fb464d32 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:22.717427 30387 tablet_service.cc:1467] Processing CreateTablet for tablet 15f40a19911b40fb980eead07bcbd106 (DEFAULT_TABLE table=rebalance_test_table_4 [id=979bbb7a065445c89d4cf2f3e28ed0d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.717633 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15f40a19911b40fb980eead07bcbd106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.717821 31302 tablet_bootstrap.cc:492] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:22.717875 31302 ts_tablet_manager.cc:1367] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:22.718163 31302 raft_consensus.cc:348] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:22.718241 31302 raft_consensus.cc:374] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.718278 31302 raft_consensus.cc:729] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.718349 31302 consensus_queue.cc:260] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:22.718567 31307 tablet_bootstrap.cc:492] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:22.718570 31302 ts_tablet_manager.cc:1398] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:22.719441 31307 tablet_bootstrap.cc:654] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.720731 31307 tablet_bootstrap.cc:492] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:22.720876 31307 ts_tablet_manager.cc:1367] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:22.720892 29994 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3e51eb67b947038487dc0c7782d0b6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=979bbb7a065445c89d4cf2f3e28ed0d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.721154 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3e51eb67b947038487dc0c7782d0b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.721222 31303 tablet_bootstrap.cc:492] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:22.721350 31307 raft_consensus.cc:348] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:22.721451 31307 raft_consensus.cc:374] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.721482 31307 raft_consensus.cc:729] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.721590 31307 consensus_queue.cc:260] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:22.721860 31307 ts_tablet_manager.cc:1398] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:22.722054 31307 tablet_bootstrap.cc:492] T 15f40a19911b40fb980eead07bcbd106 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:22.722579 31303 tablet_bootstrap.cc:654] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.722927 31307 tablet_bootstrap.cc:654] T 15f40a19911b40fb980eead07bcbd106 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.723732 31303 tablet_bootstrap.cc:492] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:22.723817 31303 ts_tablet_manager.cc:1367] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:22.724012 31307 tablet_bootstrap.cc:492] T 15f40a19911b40fb980eead07bcbd106 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:22.724136 31307 ts_tablet_manager.cc:1367] T 15f40a19911b40fb980eead07bcbd106 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:22.724500 31303 raft_consensus.cc:348] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.724619 31303 raft_consensus.cc:374] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.724581 31307 raft_consensus.cc:348] T 15f40a19911b40fb980eead07bcbd106 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:22.724654 31303 raft_consensus.cc:729] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.724678 31307 raft_consensus.cc:374] T 15f40a19911b40fb980eead07bcbd106 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.724710 31307 raft_consensus.cc:729] T 15f40a19911b40fb980eead07bcbd106 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.724771 31303 consensus_queue.cc:260] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.724871 31307 consensus_queue.cc:260] T 15f40a19911b40fb980eead07bcbd106 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:22.725047 31301 tablet_bootstrap.cc:492] T 6f3e51eb67b947038487dc0c7782d0b6 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:22.725055 31303 ts_tablet_manager.cc:1398] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:22.725201 31303 tablet_bootstrap.cc:492] T 6f3e51eb67b947038487dc0c7782d0b6 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:22.725793 31303 tablet_bootstrap.cc:654] T 6f3e51eb67b947038487dc0c7782d0b6 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.726089 31301 tablet_bootstrap.cc:654] T 6f3e51eb67b947038487dc0c7782d0b6 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.726099 31307 ts_tablet_manager.cc:1398] T 15f40a19911b40fb980eead07bcbd106 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:22.720892 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 15f40a19911b40fb980eead07bcbd106 (DEFAULT_TABLE table=rebalance_test_table_4 [id=979bbb7a065445c89d4cf2f3e28ed0d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.727212 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15f40a19911b40fb980eead07bcbd106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.727236 31301 tablet_bootstrap.cc:492] T 6f3e51eb67b947038487dc0c7782d0b6 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:22.727298 31301 ts_tablet_manager.cc:1367] T 6f3e51eb67b947038487dc0c7782d0b6 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:22.727718 31301 raft_consensus.cc:348] T 6f3e51eb67b947038487dc0c7782d0b6 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:22.727807 31301 raft_consensus.cc:374] T 6f3e51eb67b947038487dc0c7782d0b6 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.727833 31301 raft_consensus.cc:729] T 6f3e51eb67b947038487dc0c7782d0b6 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.727948 31301 consensus_queue.cc:260] T 6f3e51eb67b947038487dc0c7782d0b6 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:22.728286 31301 ts_tablet_manager.cc:1398] T 6f3e51eb67b947038487dc0c7782d0b6 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:22.729337 31301 tablet_bootstrap.cc:492] T 15f40a19911b40fb980eead07bcbd106 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:22.730345 31301 tablet_bootstrap.cc:654] T 15f40a19911b40fb980eead07bcbd106 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.730464 31303 tablet_bootstrap.cc:492] T 6f3e51eb67b947038487dc0c7782d0b6 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:22.730552 31303 ts_tablet_manager.cc:1367] T 6f3e51eb67b947038487dc0c7782d0b6 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:22.731011 31303 raft_consensus.cc:348] T 6f3e51eb67b947038487dc0c7782d0b6 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:22.731119 31303 raft_consensus.cc:374] T 6f3e51eb67b947038487dc0c7782d0b6 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.731160 31303 raft_consensus.cc:729] T 6f3e51eb67b947038487dc0c7782d0b6 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.731272 31303 consensus_queue.cc:260] T 6f3e51eb67b947038487dc0c7782d0b6 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:22.731557 31303 ts_tablet_manager.cc:1398] T 6f3e51eb67b947038487dc0c7782d0b6 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:22.731746 31301 tablet_bootstrap.cc:492] T 15f40a19911b40fb980eead07bcbd106 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:22.731856 31301 ts_tablet_manager.cc:1367] T 15f40a19911b40fb980eead07bcbd106 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:22.732263 31301 raft_consensus.cc:348] T 15f40a19911b40fb980eead07bcbd106 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:22.732350 31301 raft_consensus.cc:374] T 15f40a19911b40fb980eead07bcbd106 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.732371 31301 raft_consensus.cc:729] T 15f40a19911b40fb980eead07bcbd106 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.732508 31301 consensus_queue.cc:260] T 15f40a19911b40fb980eead07bcbd106 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:22.732811 31301 ts_tablet_manager.cc:1398] T 15f40a19911b40fb980eead07bcbd106 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:22.752074 31061 raft_consensus.cc:482] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:22.752172 31061 raft_consensus.cc:504] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.752563 31061 leader_election.cc:288] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:22.753389 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6731d4b3824d3eaeba37e2fb464d32" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:22.753538 30276 raft_consensus.cc:2454] T ef6731d4b3824d3eaeba37e2fb464d32 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:22.753858 30081 leader_election.cc:302] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:22.754032 31061 raft_consensus.cc:2781] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:22.754096 31061 raft_consensus.cc:482] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:22.754132 31061 raft_consensus.cc:3037] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.755113 31061 raft_consensus.cc:504] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.755425 31061 leader_election.cc:288] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:22.755764 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6731d4b3824d3eaeba37e2fb464d32" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:22.755890 30276 raft_consensus.cc:3037] T ef6731d4b3824d3eaeba37e2fb464d32 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.756695 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6731d4b3824d3eaeba37e2fb464d32" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:22.756742 30015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6731d4b3824d3eaeba37e2fb464d32" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:22.756846 30014 raft_consensus.cc:2454] T ef6731d4b3824d3eaeba37e2fb464d32 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:22.756844 30276 raft_consensus.cc:2454] T ef6731d4b3824d3eaeba37e2fb464d32 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:22.757093 30081 leader_election.cc:302] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:22.757265 31061 raft_consensus.cc:2781] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:22.757477 31061 raft_consensus.cc:686] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:22.757619 31061 consensus_queue.cc:237] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:22.759214 29868 catalog_manager.cc:5484] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:22.771342 31238 raft_consensus.cc:482] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:22.771451 31238 raft_consensus.cc:504] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:22.771917 31238 leader_election.cc:288] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:22.772290 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3e51eb67b947038487dc0c7782d0b6" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:22.772419 30146 raft_consensus.cc:2454] T 6f3e51eb67b947038487dc0c7782d0b6 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:22.772712 30344 leader_election.cc:302] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:22.772878 31238 raft_consensus.cc:2781] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:22.772943 31238 raft_consensus.cc:482] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:22.772990 31238 raft_consensus.cc:3037] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.774199 31238 raft_consensus.cc:504] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:22.774542 31238 leader_election.cc:288] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:22.775086 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3e51eb67b947038487dc0c7782d0b6" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:22.775197 30146 raft_consensus.cc:3037] T 6f3e51eb67b947038487dc0c7782d0b6 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.776104 30146 raft_consensus.cc:2454] T 6f3e51eb67b947038487dc0c7782d0b6 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:22.776199 30015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3e51eb67b947038487dc0c7782d0b6" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:22.776279 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3e51eb67b947038487dc0c7782d0b6" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:22.776361 30015 raft_consensus.cc:2454] T 6f3e51eb67b947038487dc0c7782d0b6 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:22.776484 30344 leader_election.cc:302] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:22.776647 31238 raft_consensus.cc:2781] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:22.776746 31238 raft_consensus.cc:686] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:22.776885 31238 consensus_queue.cc:237] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:22.778539 29868 catalog_manager.cc:5484] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:23.054567 31114 raft_consensus.cc:482] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:23.054680 31114 raft_consensus.cc:504] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:23.055060 31114 leader_election.cc:288] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:23.055416 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15f40a19911b40fb980eead07bcbd106" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:23.055414 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15f40a19911b40fb980eead07bcbd106" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:23.055541 30014 raft_consensus.cc:2454] T 15f40a19911b40fb980eead07bcbd106 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:23.055554 30407 raft_consensus.cc:2454] T 15f40a19911b40fb980eead07bcbd106 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:23.055822 30212 leader_election.cc:302] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:23.055979 31114 raft_consensus.cc:2781] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:23.056038 31114 raft_consensus.cc:482] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:23.056087 31114 raft_consensus.cc:3037] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.057029 31114 raft_consensus.cc:504] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:23.057323 31114 leader_election.cc:288] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:23.057608 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15f40a19911b40fb980eead07bcbd106" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:23.057614 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15f40a19911b40fb980eead07bcbd106" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:23.057698 30014 raft_consensus.cc:3037] T 15f40a19911b40fb980eead07bcbd106 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.057703 30407 raft_consensus.cc:3037] T 15f40a19911b40fb980eead07bcbd106 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.058470 30014 raft_consensus.cc:2454] T 15f40a19911b40fb980eead07bcbd106 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:23.058480 30407 raft_consensus.cc:2454] T 15f40a19911b40fb980eead07bcbd106 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:23.058719 30210 leader_election.cc:302] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:23.058871 31114 raft_consensus.cc:2781] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:23.058962 31114 raft_consensus.cc:686] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:23.059093 31114 consensus_queue.cc:237] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:23.060688 29868 catalog_manager.cc:5484] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:23.154984 31061 consensus_queue.cc:1035] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:23.157481 30014 raft_consensus.cc:3037] T ef6731d4b3824d3eaeba37e2fb464d32 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.158883 31061 consensus_queue.cc:1035] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:23.189877 31320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:23.190073 31320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:23.194624 31320 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 14:09:23.234815 30014 raft_consensus.cc:3037] T 6f3e51eb67b947038487dc0c7782d0b6 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.236479 31238 consensus_queue.cc:1035] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:23.239692 31076 consensus_queue.cc:1035] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:23.275175 30014 raft_consensus.cc:1261] T 15f40a19911b40fb980eead07bcbd106 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:23.278309 31236 consensus_queue.cc:1035] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:23.280957 30407 raft_consensus.cc:1261] T 15f40a19911b40fb980eead07bcbd106 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:23.282799 31114 consensus_queue.cc:1035] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 102.023 ms
time per row: 0.0510117 ms
I20240501 14:09:23.337847 29868 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:23.343116 30255 tablet_service.cc:1467] Processing CreateTablet for tablet 5518112449d9448e98f5c779f1fa3248 (DEFAULT_TABLE table=rebalance_test_table_5 [id=440878c63b0f4706b34fd42316991270]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.343196 30387 tablet_service.cc:1467] Processing CreateTablet for tablet 5518112449d9448e98f5c779f1fa3248 (DEFAULT_TABLE table=rebalance_test_table_5 [id=440878c63b0f4706b34fd42316991270]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.343423 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5518112449d9448e98f5c779f1fa3248. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.343469 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5518112449d9448e98f5c779f1fa3248. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.344511 30256 tablet_service.cc:1467] Processing CreateTablet for tablet 3a04e958f3d049139ff74202611af725 (DEFAULT_TABLE table=rebalance_test_table_5 [id=440878c63b0f4706b34fd42316991270]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.344645 29993 tablet_service.cc:1467] Processing CreateTablet for tablet ee998c3fcf3449d5845ac1c950f326d7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=440878c63b0f4706b34fd42316991270]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.344775 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a04e958f3d049139ff74202611af725. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.344892 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee998c3fcf3449d5845ac1c950f326d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.345988 30124 tablet_service.cc:1467] Processing CreateTablet for tablet 5518112449d9448e98f5c779f1fa3248 (DEFAULT_TABLE table=rebalance_test_table_5 [id=440878c63b0f4706b34fd42316991270]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.346081 31380 tablet_bootstrap.cc:492] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:23.346248 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5518112449d9448e98f5c779f1fa3248. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.346819 30126 tablet_service.cc:1467] Processing CreateTablet for tablet ee998c3fcf3449d5845ac1c950f326d7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=440878c63b0f4706b34fd42316991270]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.346997 31380 tablet_bootstrap.cc:654] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.347061 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee998c3fcf3449d5845ac1c950f326d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.347296 30257 tablet_service.cc:1467] Processing CreateTablet for tablet ee998c3fcf3449d5845ac1c950f326d7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=440878c63b0f4706b34fd42316991270]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.347497 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee998c3fcf3449d5845ac1c950f326d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.347642 31379 tablet_bootstrap.cc:492] T 5518112449d9448e98f5c779f1fa3248 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:23.348532 31379 tablet_bootstrap.cc:654] T 5518112449d9448e98f5c779f1fa3248 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.348511 30125 tablet_service.cc:1467] Processing CreateTablet for tablet 3a04e958f3d049139ff74202611af725 (DEFAULT_TABLE table=rebalance_test_table_5 [id=440878c63b0f4706b34fd42316991270]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.348733 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a04e958f3d049139ff74202611af725. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.349633 31379 tablet_bootstrap.cc:492] T 5518112449d9448e98f5c779f1fa3248 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:23.349633 31383 tablet_bootstrap.cc:492] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:23.349714 31379 ts_tablet_manager.cc:1367] T 5518112449d9448e98f5c779f1fa3248 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:23.350131 31379 raft_consensus.cc:348] T 5518112449d9448e98f5c779f1fa3248 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:23.350250 31379 raft_consensus.cc:374] T 5518112449d9448e98f5c779f1fa3248 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.350283 31379 raft_consensus.cc:729] T 5518112449d9448e98f5c779f1fa3248 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.350458 31383 tablet_bootstrap.cc:654] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.350407 31379 consensus_queue.cc:260] T 5518112449d9448e98f5c779f1fa3248 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:23.350709 31379 ts_tablet_manager.cc:1398] T 5518112449d9448e98f5c779f1fa3248 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.350814 31381 tablet_bootstrap.cc:492] T ee998c3fcf3449d5845ac1c950f326d7 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:23.350831 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 3a04e958f3d049139ff74202611af725 (DEFAULT_TABLE table=rebalance_test_table_5 [id=440878c63b0f4706b34fd42316991270]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.351070 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a04e958f3d049139ff74202611af725. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.351410 31383 tablet_bootstrap.cc:492] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:23.351502 31383 ts_tablet_manager.cc:1367] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:23.351845 31383 raft_consensus.cc:348] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:23.351922 31383 raft_consensus.cc:374] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.351948 31383 raft_consensus.cc:729] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.352037 31383 consensus_queue.cc:260] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:23.352208 31383 ts_tablet_manager.cc:1398] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:23.352449 31383 tablet_bootstrap.cc:492] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:23.353255 31383 tablet_bootstrap.cc:654] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.353314 31379 tablet_bootstrap.cc:492] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:23.354085 31381 tablet_bootstrap.cc:654] T ee998c3fcf3449d5845ac1c950f326d7 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.354120 31379 tablet_bootstrap.cc:654] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.354230 31383 tablet_bootstrap.cc:492] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:23.354307 31383 ts_tablet_manager.cc:1367] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:23.354707 31383 raft_consensus.cc:348] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:23.354805 31383 raft_consensus.cc:374] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.354838 31383 raft_consensus.cc:729] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.354964 31383 consensus_queue.cc:260] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:23.355244 31383 ts_tablet_manager.cc:1398] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.355402 31383 tablet_bootstrap.cc:492] T 3a04e958f3d049139ff74202611af725 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:23.355618 31380 tablet_bootstrap.cc:492] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:23.355705 31380 ts_tablet_manager.cc:1367] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 14:09:23.356120 31380 raft_consensus.cc:348] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:23.356230 31383 tablet_bootstrap.cc:654] T 3a04e958f3d049139ff74202611af725 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.356276 31380 raft_consensus.cc:374] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.356313 31380 raft_consensus.cc:729] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.356437 31380 consensus_queue.cc:260] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:23.356727 31380 ts_tablet_manager.cc:1398] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.357220 31383 tablet_bootstrap.cc:492] T 3a04e958f3d049139ff74202611af725 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:23.357295 31383 ts_tablet_manager.cc:1367] T 3a04e958f3d049139ff74202611af725 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:23.357357 31380 tablet_bootstrap.cc:492] T 3a04e958f3d049139ff74202611af725 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:23.357357 31379 tablet_bootstrap.cc:492] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:23.357426 31379 ts_tablet_manager.cc:1367] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:23.357803 31383 raft_consensus.cc:348] T 3a04e958f3d049139ff74202611af725 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:23.357888 31383 raft_consensus.cc:374] T 3a04e958f3d049139ff74202611af725 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.357846 31379 raft_consensus.cc:348] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:23.357919 31383 raft_consensus.cc:729] T 3a04e958f3d049139ff74202611af725 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.357971 31379 raft_consensus.cc:374] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.358002 31379 raft_consensus.cc:729] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.358111 31379 consensus_queue.cc:260] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:23.358100 31383 consensus_queue.cc:260] T 3a04e958f3d049139ff74202611af725 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:23.358397 31380 tablet_bootstrap.cc:654] T 3a04e958f3d049139ff74202611af725 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.358399 31383 ts_tablet_manager.cc:1398] T 3a04e958f3d049139ff74202611af725 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:23.358422 31379 ts_tablet_manager.cc:1398] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:23.359323 31381 tablet_bootstrap.cc:492] T ee998c3fcf3449d5845ac1c950f326d7 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:23.359383 31380 tablet_bootstrap.cc:492] T 3a04e958f3d049139ff74202611af725 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:23.359467 31380 ts_tablet_manager.cc:1367] T 3a04e958f3d049139ff74202611af725 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:23.359498 31381 ts_tablet_manager.cc:1367] T ee998c3fcf3449d5845ac1c950f326d7 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:23.359899 31380 raft_consensus.cc:348] T 3a04e958f3d049139ff74202611af725 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:23.359997 31380 raft_consensus.cc:374] T 3a04e958f3d049139ff74202611af725 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.360036 31380 raft_consensus.cc:729] T 3a04e958f3d049139ff74202611af725 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.360174 31380 consensus_queue.cc:260] T 3a04e958f3d049139ff74202611af725 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:23.360168 31381 raft_consensus.cc:348] T ee998c3fcf3449d5845ac1c950f326d7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:23.360404 31381 raft_consensus.cc:374] T ee998c3fcf3449d5845ac1c950f326d7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.360472 31380 ts_tablet_manager.cc:1398] T 3a04e958f3d049139ff74202611af725 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.360484 31381 raft_consensus.cc:729] T ee998c3fcf3449d5845ac1c950f326d7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.360694 31380 tablet_bootstrap.cc:492] T ee998c3fcf3449d5845ac1c950f326d7 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:23.360687 31381 consensus_queue.cc:260] T ee998c3fcf3449d5845ac1c950f326d7 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:23.361102 31381 ts_tablet_manager.cc:1398] T ee998c3fcf3449d5845ac1c950f326d7 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.361511 31380 tablet_bootstrap.cc:654] T ee998c3fcf3449d5845ac1c950f326d7 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.362922 31380 tablet_bootstrap.cc:492] T ee998c3fcf3449d5845ac1c950f326d7 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:23.363013 31380 ts_tablet_manager.cc:1367] T ee998c3fcf3449d5845ac1c950f326d7 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:23.363420 31380 raft_consensus.cc:348] T ee998c3fcf3449d5845ac1c950f326d7 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:23.363617 31380 raft_consensus.cc:374] T ee998c3fcf3449d5845ac1c950f326d7 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.363641 31380 raft_consensus.cc:729] T ee998c3fcf3449d5845ac1c950f326d7 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.363757 31380 consensus_queue.cc:260] T ee998c3fcf3449d5845ac1c950f326d7 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:23.363960 31380 ts_tablet_manager.cc:1398] T ee998c3fcf3449d5845ac1c950f326d7 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.375185 31061 raft_consensus.cc:482] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:23.375265 31061 raft_consensus.cc:504] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:23.375573 31061 leader_election.cc:288] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:23.375861 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5518112449d9448e98f5c779f1fa3248" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:23.375861 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5518112449d9448e98f5c779f1fa3248" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:23.375977 30276 raft_consensus.cc:2454] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:23.375981 30407 raft_consensus.cc:2454] T 5518112449d9448e98f5c779f1fa3248 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:23.376279 30079 leader_election.cc:302] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:23.376420 31061 raft_consensus.cc:2781] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:23.376474 31061 raft_consensus.cc:482] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:23.376510 31061 raft_consensus.cc:3037] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.377166 31061 raft_consensus.cc:504] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:23.377454 31061 leader_election.cc:288] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:23.377732 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5518112449d9448e98f5c779f1fa3248" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:23.377737 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5518112449d9448e98f5c779f1fa3248" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:23.377820 30276 raft_consensus.cc:3037] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.377830 30407 raft_consensus.cc:3037] T 5518112449d9448e98f5c779f1fa3248 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.378571 30407 raft_consensus.cc:2454] T 5518112449d9448e98f5c779f1fa3248 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:23.378751 30276 raft_consensus.cc:2454] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:23.378829 30079 leader_election.cc:302] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:23.378991 31061 raft_consensus.cc:2781] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:23.379087 31061 raft_consensus.cc:686] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:23.379225 31061 consensus_queue.cc:237] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:23.380582 29868 catalog_manager.cc:5484] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:23.457568 31374 raft_consensus.cc:482] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:23.457687 31374 raft_consensus.cc:504] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:23.458067 31374 leader_election.cc:288] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:23.458448 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a04e958f3d049139ff74202611af725" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:23.458482 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a04e958f3d049139ff74202611af725" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:23.458585 30276 raft_consensus.cc:2454] T 3a04e958f3d049139ff74202611af725 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:23.458616 30146 raft_consensus.cc:2454] T 3a04e958f3d049139ff74202611af725 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:23.458856 30343 leader_election.cc:302] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:23.459007 31374 raft_consensus.cc:2781] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:23.459069 31374 raft_consensus.cc:482] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:23.459108 31374 raft_consensus.cc:3037] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.459920 31374 raft_consensus.cc:504] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:23.460261 31374 leader_election.cc:288] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:23.460536 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a04e958f3d049139ff74202611af725" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:23.460598 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a04e958f3d049139ff74202611af725" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:23.460650 30276 raft_consensus.cc:3037] T 3a04e958f3d049139ff74202611af725 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.460718 30146 raft_consensus.cc:3037] T 3a04e958f3d049139ff74202611af725 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.461364 30276 raft_consensus.cc:2454] T 3a04e958f3d049139ff74202611af725 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:23.461623 30343 leader_election.cc:302] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:23.461776 31374 raft_consensus.cc:2781] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:23.461853 31374 raft_consensus.cc:686] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:23.461992 31374 consensus_queue.cc:237] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:23.463649 29868 catalog_manager.cc:5484] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:23.480873 30146 raft_consensus.cc:2454] T 3a04e958f3d049139ff74202611af725 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:23.655251 31061 raft_consensus.cc:482] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:23.655381 31061 raft_consensus.cc:504] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:23.655745 31061 leader_election.cc:288] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:23.656175 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee998c3fcf3449d5845ac1c950f326d7" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:23.656172 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee998c3fcf3449d5845ac1c950f326d7" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:23.656368 30276 raft_consensus.cc:2454] T ee998c3fcf3449d5845ac1c950f326d7 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:23.656368 30013 raft_consensus.cc:2454] T ee998c3fcf3449d5845ac1c950f326d7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:23.656635 30081 leader_election.cc:302] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:23.656805 31061 raft_consensus.cc:2781] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:23.656863 31061 raft_consensus.cc:482] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:23.656914 31061 raft_consensus.cc:3037] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.657688 31061 raft_consensus.cc:504] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:23.657984 31061 leader_election.cc:288] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:23.658298 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee998c3fcf3449d5845ac1c950f326d7" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:23.658387 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee998c3fcf3449d5845ac1c950f326d7" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:23.658421 30013 raft_consensus.cc:3037] T ee998c3fcf3449d5845ac1c950f326d7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.658497 30276 raft_consensus.cc:3037] T ee998c3fcf3449d5845ac1c950f326d7 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.659348 30276 raft_consensus.cc:2454] T ee998c3fcf3449d5845ac1c950f326d7 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:23.659348 30013 raft_consensus.cc:2454] T ee998c3fcf3449d5845ac1c950f326d7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:23.659621 30081 leader_election.cc:302] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:23.659762 31061 raft_consensus.cc:2781] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:23.659845 31061 raft_consensus.cc:686] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:23.659979 31061 consensus_queue.cc:237] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:23.661686 29868 catalog_manager.cc:5484] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
W20240501 14:09:23.815062 31395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:23.815290 31395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:23.818192 31395 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 14:09:23.859467 31061 consensus_queue.cc:1035] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:23.863932 31061 consensus_queue.cc:1035] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:23.904908 30276 raft_consensus.cc:1261] T ee998c3fcf3449d5845ac1c950f326d7 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:23.904908 30013 raft_consensus.cc:1261] T ee998c3fcf3449d5845ac1c950f326d7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:23.905498 31315 consensus_queue.cc:1035] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:23.905643 31061 consensus_queue.cc:1035] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:23.908418 30277 raft_consensus.cc:1261] T 3a04e958f3d049139ff74202611af725 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:23.909037 30146 raft_consensus.cc:1261] T 3a04e958f3d049139ff74202611af725 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:23.909417 31238 consensus_queue.cc:1035] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:23.909783 31238 consensus_queue.cc:1035] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:23.920315 31431 mvcc.cc:205] Tried to move back new op lower bound from 7022889221749940224 to 7022889220752662528. Current Snapshot: MvccSnapshot[applied={T|T < 7022889221749940224}]
INSERT report
rows total: 2000
time total: 106.269 ms
time per row: 0.0531344 ms
I20240501 14:09:23.965449 29872 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:23.970567 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 8c5a380ad2624291b9770c3f1633c14d (DEFAULT_TABLE table=rebalance_test_table_6 [id=a49ff4b9b5fe43dea63424b7e0856b72]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.970654 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 8c5a380ad2624291b9770c3f1633c14d (DEFAULT_TABLE table=rebalance_test_table_6 [id=a49ff4b9b5fe43dea63424b7e0856b72]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.970891 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c5a380ad2624291b9770c3f1633c14d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.970891 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c5a380ad2624291b9770c3f1633c14d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.971355 30387 tablet_service.cc:1467] Processing CreateTablet for tablet 4f5314fbd59c4443974a9c304ba9b1a7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=a49ff4b9b5fe43dea63424b7e0856b72]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.971547 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f5314fbd59c4443974a9c304ba9b1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.973126 31451 tablet_bootstrap.cc:492] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:23.973429 30257 tablet_service.cc:1467] Processing CreateTablet for tablet 4f5314fbd59c4443974a9c304ba9b1a7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=a49ff4b9b5fe43dea63424b7e0856b72]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.973536 30386 tablet_service.cc:1467] Processing CreateTablet for tablet f4b8334e04994f5f8dc96eb91b944a47 (DEFAULT_TABLE table=rebalance_test_table_6 [id=a49ff4b9b5fe43dea63424b7e0856b72]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.973676 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f5314fbd59c4443974a9c304ba9b1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.973711 30386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4b8334e04994f5f8dc96eb91b944a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.973870 31451 tablet_bootstrap.cc:654] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.975926 30125 tablet_service.cc:1467] Processing CreateTablet for tablet 8c5a380ad2624291b9770c3f1633c14d (DEFAULT_TABLE table=rebalance_test_table_6 [id=a49ff4b9b5fe43dea63424b7e0856b72]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.975975 30126 tablet_service.cc:1467] Processing CreateTablet for tablet f4b8334e04994f5f8dc96eb91b944a47 (DEFAULT_TABLE table=rebalance_test_table_6 [id=a49ff4b9b5fe43dea63424b7e0856b72]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.976214 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c5a380ad2624291b9770c3f1633c14d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.976363 29994 tablet_service.cc:1467] Processing CreateTablet for tablet 4f5314fbd59c4443974a9c304ba9b1a7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=a49ff4b9b5fe43dea63424b7e0856b72]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.976490 29995 tablet_service.cc:1467] Processing CreateTablet for tablet f4b8334e04994f5f8dc96eb91b944a47 (DEFAULT_TABLE table=rebalance_test_table_6 [id=a49ff4b9b5fe43dea63424b7e0856b72]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.976593 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f5314fbd59c4443974a9c304ba9b1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.976696 31451 tablet_bootstrap.cc:492] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:23.976699 29995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4b8334e04994f5f8dc96eb91b944a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.976763 31451 ts_tablet_manager.cc:1367] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:23.977237 31451 raft_consensus.cc:348] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:23.977387 31451 raft_consensus.cc:374] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.977445 31451 raft_consensus.cc:729] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.977470 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4b8334e04994f5f8dc96eb91b944a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.977577 31451 consensus_queue.cc:260] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:23.978315 31451 ts_tablet_manager.cc:1398] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:23.978482 31451 tablet_bootstrap.cc:492] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:23.979321 31451 tablet_bootstrap.cc:654] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.979638 31453 tablet_bootstrap.cc:492] T 4f5314fbd59c4443974a9c304ba9b1a7 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:23.980429 31453 tablet_bootstrap.cc:654] T 4f5314fbd59c4443974a9c304ba9b1a7 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.981475 31453 tablet_bootstrap.cc:492] T 4f5314fbd59c4443974a9c304ba9b1a7 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:23.981554 31453 ts_tablet_manager.cc:1367] T 4f5314fbd59c4443974a9c304ba9b1a7 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:23.981988 31453 raft_consensus.cc:348] T 4f5314fbd59c4443974a9c304ba9b1a7 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:23.982112 31453 raft_consensus.cc:374] T 4f5314fbd59c4443974a9c304ba9b1a7 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.982144 31453 raft_consensus.cc:729] T 4f5314fbd59c4443974a9c304ba9b1a7 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.982275 31453 consensus_queue.cc:260] T 4f5314fbd59c4443974a9c304ba9b1a7 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:23.982368 31451 tablet_bootstrap.cc:492] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:23.982537 31453 ts_tablet_manager.cc:1398] T 4f5314fbd59c4443974a9c304ba9b1a7 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.982430 31451 ts_tablet_manager.cc:1367] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:23.983246 31451 raft_consensus.cc:348] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:23.983326 31451 raft_consensus.cc:374] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.983358 31451 raft_consensus.cc:729] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.983628 31451 consensus_queue.cc:260] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:23.983909 31451 ts_tablet_manager.cc:1398] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.984604 31451 tablet_bootstrap.cc:492] T f4b8334e04994f5f8dc96eb91b944a47 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:23.985466 31451 tablet_bootstrap.cc:654] T f4b8334e04994f5f8dc96eb91b944a47 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.986773 31451 tablet_bootstrap.cc:492] T f4b8334e04994f5f8dc96eb91b944a47 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:23.986845 31451 ts_tablet_manager.cc:1367] T f4b8334e04994f5f8dc96eb91b944a47 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:23.987139 31451 raft_consensus.cc:348] T f4b8334e04994f5f8dc96eb91b944a47 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:23.987215 31451 raft_consensus.cc:374] T f4b8334e04994f5f8dc96eb91b944a47 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.987234 31451 raft_consensus.cc:729] T f4b8334e04994f5f8dc96eb91b944a47 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.987337 31451 consensus_queue.cc:260] T f4b8334e04994f5f8dc96eb91b944a47 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:23.987607 31451 ts_tablet_manager.cc:1398] T f4b8334e04994f5f8dc96eb91b944a47 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:23.992415 31454 tablet_bootstrap.cc:492] T 8c5a380ad2624291b9770c3f1633c14d P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:23.993197 31454 tablet_bootstrap.cc:654] T 8c5a380ad2624291b9770c3f1633c14d P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.994297 31458 tablet_bootstrap.cc:492] T 8c5a380ad2624291b9770c3f1633c14d P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:23.994951 31454 tablet_bootstrap.cc:492] T 8c5a380ad2624291b9770c3f1633c14d P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:23.995049 31454 ts_tablet_manager.cc:1367] T 8c5a380ad2624291b9770c3f1633c14d P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:23.995079 31458 tablet_bootstrap.cc:654] T 8c5a380ad2624291b9770c3f1633c14d P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.995496 31454 raft_consensus.cc:348] T 8c5a380ad2624291b9770c3f1633c14d P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:23.995599 31454 raft_consensus.cc:374] T 8c5a380ad2624291b9770c3f1633c14d P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.995633 31454 raft_consensus.cc:729] T 8c5a380ad2624291b9770c3f1633c14d P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.995752 31454 consensus_queue.cc:260] T 8c5a380ad2624291b9770c3f1633c14d P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:23.996034 31454 ts_tablet_manager.cc:1398] T 8c5a380ad2624291b9770c3f1633c14d P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.996047 31458 tablet_bootstrap.cc:492] T 8c5a380ad2624291b9770c3f1633c14d P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:23.996124 31458 ts_tablet_manager.cc:1367] T 8c5a380ad2624291b9770c3f1633c14d P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:23.996302 31454 tablet_bootstrap.cc:492] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:23.996532 31458 raft_consensus.cc:348] T 8c5a380ad2624291b9770c3f1633c14d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:23.996637 31458 raft_consensus.cc:374] T 8c5a380ad2624291b9770c3f1633c14d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.996680 31458 raft_consensus.cc:729] T 8c5a380ad2624291b9770c3f1633c14d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.996824 31458 consensus_queue.cc:260] T 8c5a380ad2624291b9770c3f1633c14d P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:23.997113 31454 tablet_bootstrap.cc:654] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.997119 31458 ts_tablet_manager.cc:1398] T 8c5a380ad2624291b9770c3f1633c14d P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.997293 31458 tablet_bootstrap.cc:492] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:23.998049 31458 tablet_bootstrap.cc:654] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.998427 31454 tablet_bootstrap.cc:492] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:23.998504 31454 ts_tablet_manager.cc:1367] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:23.998986 31454 raft_consensus.cc:348] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:23.999070 31458 tablet_bootstrap.cc:492] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:23.999142 31458 ts_tablet_manager.cc:1367] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:23.999073 31454 raft_consensus.cc:374] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.999179 31454 raft_consensus.cc:729] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.999326 31454 consensus_queue.cc:260] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:23.999583 31458 raft_consensus.cc:348] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:23.999696 31458 raft_consensus.cc:374] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.999732 31458 raft_consensus.cc:729] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.999742 31454 ts_tablet_manager.cc:1398] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.999855 31458 consensus_queue.cc:260] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:24.000120 31458 ts_tablet_manager.cc:1398] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:24.000349 31458 tablet_bootstrap.cc:492] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:24.001153 31458 tablet_bootstrap.cc:654] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.002635 31458 tablet_bootstrap.cc:492] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:24.002714 31458 ts_tablet_manager.cc:1367] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:24.003048 31458 raft_consensus.cc:348] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:24.003124 31458 raft_consensus.cc:374] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.003156 31458 raft_consensus.cc:729] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.003274 31458 consensus_queue.cc:260] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:24.003468 31458 ts_tablet_manager.cc:1398] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:24.019335 31238 raft_consensus.cc:482] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:24.019472 31238 raft_consensus.cc:504] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:24.019876 31238 leader_election.cc:288] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:24.020268 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c5a380ad2624291b9770c3f1633c14d" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:24.020411 30013 raft_consensus.cc:2454] T 8c5a380ad2624291b9770c3f1633c14d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:24.020448 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c5a380ad2624291b9770c3f1633c14d" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:24.020573 30146 raft_consensus.cc:2454] T 8c5a380ad2624291b9770c3f1633c14d P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:24.020679 30343 leader_election.cc:302] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:24.020866 31238 raft_consensus.cc:2781] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:24.020921 31238 raft_consensus.cc:482] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:24.020956 31238 raft_consensus.cc:3037] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.021770 31238 raft_consensus.cc:504] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:24.022094 31238 leader_election.cc:288] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:24.022382 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c5a380ad2624291b9770c3f1633c14d" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:24.022477 30146 raft_consensus.cc:3037] T 8c5a380ad2624291b9770c3f1633c14d P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.022477 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c5a380ad2624291b9770c3f1633c14d" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:24.022578 30013 raft_consensus.cc:3037] T 8c5a380ad2624291b9770c3f1633c14d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.023269 30146 raft_consensus.cc:2454] T 8c5a380ad2624291b9770c3f1633c14d P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:24.023315 30013 raft_consensus.cc:2454] T 8c5a380ad2624291b9770c3f1633c14d P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:24.023561 30343 leader_election.cc:302] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:24.023738 31238 raft_consensus.cc:2781] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:24.023815 31238 raft_consensus.cc:686] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:24.023957 31238 consensus_queue.cc:237] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:24.025439 29872 catalog_manager.cc:5484] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:24.129007 31238 raft_consensus.cc:482] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:24.129123 31238 raft_consensus.cc:504] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:24.129488 31238 leader_election.cc:288] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:24.129833 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f5314fbd59c4443974a9c304ba9b1a7" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:24.129915 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f5314fbd59c4443974a9c304ba9b1a7" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:24.129988 30276 raft_consensus.cc:2454] T 4f5314fbd59c4443974a9c304ba9b1a7 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:24.130048 30013 raft_consensus.cc:2454] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:24.130267 30343 leader_election.cc:302] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:24.130429 31238 raft_consensus.cc:2781] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:24.130496 31238 raft_consensus.cc:482] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:24.130550 31238 raft_consensus.cc:3037] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.131435 31238 raft_consensus.cc:504] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:24.131767 31238 leader_election.cc:288] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:24.132128 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f5314fbd59c4443974a9c304ba9b1a7" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:24.132128 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f5314fbd59c4443974a9c304ba9b1a7" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:24.132272 30276 raft_consensus.cc:3037] T 4f5314fbd59c4443974a9c304ba9b1a7 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.132278 30013 raft_consensus.cc:3037] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.133169 30013 raft_consensus.cc:2454] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:24.133174 30276 raft_consensus.cc:2454] T 4f5314fbd59c4443974a9c304ba9b1a7 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:24.133462 30343 leader_election.cc:302] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:24.133605 31238 raft_consensus.cc:2781] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:24.133692 31238 raft_consensus.cc:686] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:24.133822 31238 consensus_queue.cc:237] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:24.135612 29872 catalog_manager.cc:5484] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:24.238014 31019 raft_consensus.cc:482] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:24.238138 31019 raft_consensus.cc:504] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:24.238595 31019 leader_election.cc:288] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:24.239010 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b8334e04994f5f8dc96eb91b944a47" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:24.239014 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b8334e04994f5f8dc96eb91b944a47" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:24.239145 30146 raft_consensus.cc:2454] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:24.239149 30407 raft_consensus.cc:2454] T f4b8334e04994f5f8dc96eb91b944a47 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:24.239456 29951 leader_election.cc:302] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:24.239643 31019 raft_consensus.cc:2781] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:24.239694 31019 raft_consensus.cc:482] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:24.239722 31019 raft_consensus.cc:3037] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.240600 31019 raft_consensus.cc:504] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:24.240917 31019 leader_election.cc:288] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:24.241232 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b8334e04994f5f8dc96eb91b944a47" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:24.241281 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b8334e04994f5f8dc96eb91b944a47" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:24.241362 30146 raft_consensus.cc:3037] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.241365 30407 raft_consensus.cc:3037] T f4b8334e04994f5f8dc96eb91b944a47 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.242060 30407 raft_consensus.cc:2454] T f4b8334e04994f5f8dc96eb91b944a47 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:24.242149 30146 raft_consensus.cc:2454] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:24.242344 29948 leader_election.cc:302] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:24.242493 31019 raft_consensus.cc:2781] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:24.242681 31019 raft_consensus.cc:686] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:24.242830 31019 consensus_queue.cc:237] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:24.244567 29872 catalog_manager.cc:5484] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
W20240501 14:09:24.374296 31468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:24.374485 31468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:24.377421 31468 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 14:09:24.461403 30146 raft_consensus.cc:1261] T 8c5a380ad2624291b9770c3f1633c14d P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:24.461771 30013 raft_consensus.cc:1261] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:24.462229 30276 raft_consensus.cc:1261] T 4f5314fbd59c4443974a9c304ba9b1a7 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:24.462625 31238 consensus_queue.cc:1035] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:24.463413 31374 consensus_queue.cc:1035] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:24.463689 31374 consensus_queue.cc:1035] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:24.465610 30145 raft_consensus.cc:1261] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:24.465723 30407 raft_consensus.cc:1261] T f4b8334e04994f5f8dc96eb91b944a47 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:24.466684 31019 consensus_queue.cc:1035] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:24.468629 30014 raft_consensus.cc:1261] T 8c5a380ad2624291b9770c3f1633c14d P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:24.468791 31019 consensus_queue.cc:1035] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:24.471897 31238 consensus_queue.cc:1035] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:24.477690 31499 mvcc.cc:205] Tried to move back new op lower bound from 7022889224027389952 to 7022889222243442688. Current Snapshot: MvccSnapshot[applied={T|T < 7022889224027389952}]
INSERT report
rows total: 2000
time total: 113.494 ms
time per row: 0.056747 ms
I20240501 14:09:24.529732 29871 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:24.534991 29994 tablet_service.cc:1467] Processing CreateTablet for tablet 8d87b2a4f6764e20afbfb43681f58467 (DEFAULT_TABLE table=rebalance_test_table_7 [id=8f3109027b0c482d89eb86da7f7ed641]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.535077 30126 tablet_service.cc:1467] Processing CreateTablet for tablet 8d87b2a4f6764e20afbfb43681f58467 (DEFAULT_TABLE table=rebalance_test_table_7 [id=8f3109027b0c482d89eb86da7f7ed641]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.535303 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d87b2a4f6764e20afbfb43681f58467. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.535321 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d87b2a4f6764e20afbfb43681f58467. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.535709 30257 tablet_service.cc:1467] Processing CreateTablet for tablet 8d87b2a4f6764e20afbfb43681f58467 (DEFAULT_TABLE table=rebalance_test_table_7 [id=8f3109027b0c482d89eb86da7f7ed641]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.535992 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d87b2a4f6764e20afbfb43681f58467. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.537549 31527 tablet_bootstrap.cc:492] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:24.537693 30255 tablet_service.cc:1467] Processing CreateTablet for tablet fc28b2028a9246b99e8887f03180d0c3 (DEFAULT_TABLE table=rebalance_test_table_7 [id=8f3109027b0c482d89eb86da7f7ed641]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.537918 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc28b2028a9246b99e8887f03180d0c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.538164 30256 tablet_service.cc:1467] Processing CreateTablet for tablet c5d406c64b2e432a88e8c675b91dd144 (DEFAULT_TABLE table=rebalance_test_table_7 [id=8f3109027b0c482d89eb86da7f7ed641]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.538417 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d406c64b2e432a88e8c675b91dd144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.538506 31527 tablet_bootstrap.cc:654] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.539332 31528 tablet_bootstrap.cc:492] T 8d87b2a4f6764e20afbfb43681f58467 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:24.539780 29994 tablet_service.cc:1467] Processing CreateTablet for tablet c5d406c64b2e432a88e8c675b91dd144 (DEFAULT_TABLE table=rebalance_test_table_7 [id=8f3109027b0c482d89eb86da7f7ed641]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.539997 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d406c64b2e432a88e8c675b91dd144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.540184 31528 tablet_bootstrap.cc:654] T 8d87b2a4f6764e20afbfb43681f58467 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.540902 30126 tablet_service.cc:1467] Processing CreateTablet for tablet fc28b2028a9246b99e8887f03180d0c3 (DEFAULT_TABLE table=rebalance_test_table_7 [id=8f3109027b0c482d89eb86da7f7ed641]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.541126 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc28b2028a9246b99e8887f03180d0c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.541399 30387 tablet_service.cc:1467] Processing CreateTablet for tablet fc28b2028a9246b99e8887f03180d0c3 (DEFAULT_TABLE table=rebalance_test_table_7 [id=8f3109027b0c482d89eb86da7f7ed641]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.541671 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc28b2028a9246b99e8887f03180d0c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.541399 30386 tablet_service.cc:1467] Processing CreateTablet for tablet c5d406c64b2e432a88e8c675b91dd144 (DEFAULT_TABLE table=rebalance_test_table_7 [id=8f3109027b0c482d89eb86da7f7ed641]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.541872 30386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d406c64b2e432a88e8c675b91dd144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.541870 31529 tablet_bootstrap.cc:492] T 8d87b2a4f6764e20afbfb43681f58467 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:24.542719 31529 tablet_bootstrap.cc:654] T 8d87b2a4f6764e20afbfb43681f58467 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.541128 31527 tablet_bootstrap.cc:492] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:24.543448 31527 ts_tablet_manager.cc:1367] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:24.543893 31527 raft_consensus.cc:348] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:24.544003 31527 raft_consensus.cc:374] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.544039 31527 raft_consensus.cc:729] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.543982 31533 tablet_bootstrap.cc:492] T c5d406c64b2e432a88e8c675b91dd144 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:24.544179 31527 consensus_queue.cc:260] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:24.544591 31527 ts_tablet_manager.cc:1398] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:24.544847 31527 tablet_bootstrap.cc:492] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:24.545006 31528 tablet_bootstrap.cc:492] T 8d87b2a4f6764e20afbfb43681f58467 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:24.545017 31529 tablet_bootstrap.cc:492] T 8d87b2a4f6764e20afbfb43681f58467 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:24.545089 31529 ts_tablet_manager.cc:1367] T 8d87b2a4f6764e20afbfb43681f58467 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:24.545089 31528 ts_tablet_manager.cc:1367] T 8d87b2a4f6764e20afbfb43681f58467 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:24.545693 31527 tablet_bootstrap.cc:654] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.545641 31529 raft_consensus.cc:348] T 8d87b2a4f6764e20afbfb43681f58467 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:24.545853 31529 raft_consensus.cc:374] T 8d87b2a4f6764e20afbfb43681f58467 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.545935 31529 raft_consensus.cc:729] T 8d87b2a4f6764e20afbfb43681f58467 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.546003 31528 raft_consensus.cc:348] T 8d87b2a4f6764e20afbfb43681f58467 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:24.546121 31528 raft_consensus.cc:374] T 8d87b2a4f6764e20afbfb43681f58467 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.546164 31528 raft_consensus.cc:729] T 8d87b2a4f6764e20afbfb43681f58467 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.546260 31529 consensus_queue.cc:260] T 8d87b2a4f6764e20afbfb43681f58467 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:24.546308 31528 consensus_queue.cc:260] T 8d87b2a4f6764e20afbfb43681f58467 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:24.546617 31529 ts_tablet_manager.cc:1398] T 8d87b2a4f6764e20afbfb43681f58467 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:24.546778 31529 tablet_bootstrap.cc:492] T fc28b2028a9246b99e8887f03180d0c3 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:24.547124 31528 ts_tablet_manager.cc:1398] T 8d87b2a4f6764e20afbfb43681f58467 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:24.547819 31529 tablet_bootstrap.cc:654] T fc28b2028a9246b99e8887f03180d0c3 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.547960 31527 tablet_bootstrap.cc:492] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:24.548039 31527 ts_tablet_manager.cc:1367] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:24.548579 31527 raft_consensus.cc:348] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:24.548676 31527 raft_consensus.cc:374] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.548718 31527 raft_consensus.cc:729] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.549101 31533 tablet_bootstrap.cc:654] T c5d406c64b2e432a88e8c675b91dd144 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.549764 31528 tablet_bootstrap.cc:492] T c5d406c64b2e432a88e8c675b91dd144 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:24.548830 31527 consensus_queue.cc:260] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:24.550372 31527 ts_tablet_manager.cc:1398] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:24.551271 31529 tablet_bootstrap.cc:492] T fc28b2028a9246b99e8887f03180d0c3 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:24.551292 31528 tablet_bootstrap.cc:654] T c5d406c64b2e432a88e8c675b91dd144 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.551348 31529 ts_tablet_manager.cc:1367] T fc28b2028a9246b99e8887f03180d0c3 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:24.551724 31529 raft_consensus.cc:348] T fc28b2028a9246b99e8887f03180d0c3 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:24.551824 31529 raft_consensus.cc:374] T fc28b2028a9246b99e8887f03180d0c3 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.551856 31529 raft_consensus.cc:729] T fc28b2028a9246b99e8887f03180d0c3 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.551975 31529 consensus_queue.cc:260] T fc28b2028a9246b99e8887f03180d0c3 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:24.552201 31528 tablet_bootstrap.cc:492] T c5d406c64b2e432a88e8c675b91dd144 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:24.552218 31529 ts_tablet_manager.cc:1398] T fc28b2028a9246b99e8887f03180d0c3 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:24.552314 31528 ts_tablet_manager.cc:1367] T c5d406c64b2e432a88e8c675b91dd144 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:24.552399 31529 tablet_bootstrap.cc:492] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:24.552611 31528 raft_consensus.cc:348] T c5d406c64b2e432a88e8c675b91dd144 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:24.552692 31528 raft_consensus.cc:374] T c5d406c64b2e432a88e8c675b91dd144 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.552722 31528 raft_consensus.cc:729] T c5d406c64b2e432a88e8c675b91dd144 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.552830 31528 consensus_queue.cc:260] T c5d406c64b2e432a88e8c675b91dd144 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:24.553141 31528 ts_tablet_manager.cc:1398] T c5d406c64b2e432a88e8c675b91dd144 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:24.553171 31529 tablet_bootstrap.cc:654] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.553880 31533 tablet_bootstrap.cc:492] T c5d406c64b2e432a88e8c675b91dd144 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:24.554004 31533 ts_tablet_manager.cc:1367] T c5d406c64b2e432a88e8c675b91dd144 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:24.554486 31533 raft_consensus.cc:348] T c5d406c64b2e432a88e8c675b91dd144 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:24.554595 31533 raft_consensus.cc:374] T c5d406c64b2e432a88e8c675b91dd144 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.554634 31533 raft_consensus.cc:729] T c5d406c64b2e432a88e8c675b91dd144 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.554674 31529 tablet_bootstrap.cc:492] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:24.554745 31529 ts_tablet_manager.cc:1367] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:24.554776 31533 consensus_queue.cc:260] T c5d406c64b2e432a88e8c675b91dd144 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:24.555143 31529 raft_consensus.cc:348] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:24.555222 31529 raft_consensus.cc:374] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.555258 31529 raft_consensus.cc:729] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.555419 31529 consensus_queue.cc:260] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:24.555608 31533 ts_tablet_manager.cc:1398] T c5d406c64b2e432a88e8c675b91dd144 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:24.555757 31533 tablet_bootstrap.cc:492] T fc28b2028a9246b99e8887f03180d0c3 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:24.557103 31533 tablet_bootstrap.cc:654] T fc28b2028a9246b99e8887f03180d0c3 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.557220 31529 ts_tablet_manager.cc:1398] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:24.558138 31533 tablet_bootstrap.cc:492] T fc28b2028a9246b99e8887f03180d0c3 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:24.558208 31533 ts_tablet_manager.cc:1367] T fc28b2028a9246b99e8887f03180d0c3 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:24.558560 31533 raft_consensus.cc:348] T fc28b2028a9246b99e8887f03180d0c3 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:24.558665 31533 raft_consensus.cc:374] T fc28b2028a9246b99e8887f03180d0c3 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.558712 31533 raft_consensus.cc:729] T fc28b2028a9246b99e8887f03180d0c3 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.558831 31533 consensus_queue.cc:260] T fc28b2028a9246b99e8887f03180d0c3 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:24.559067 31533 ts_tablet_manager.cc:1398] T fc28b2028a9246b99e8887f03180d0c3 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:24.667570 31061 raft_consensus.cc:482] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:24.667687 31061 raft_consensus.cc:504] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:24.668083 31061 leader_election.cc:288] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:24.668495 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc28b2028a9246b99e8887f03180d0c3" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:24.668495 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc28b2028a9246b99e8887f03180d0c3" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:24.668640 30276 raft_consensus.cc:2454] T fc28b2028a9246b99e8887f03180d0c3 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:24.668640 30407 raft_consensus.cc:2454] T fc28b2028a9246b99e8887f03180d0c3 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:24.668927 30081 leader_election.cc:302] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:24.669108 31061 raft_consensus.cc:2781] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:24.669164 31061 raft_consensus.cc:482] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:24.669190 31061 raft_consensus.cc:3037] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.670058 31061 raft_consensus.cc:504] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:24.670356 31061 leader_election.cc:288] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:24.670639 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc28b2028a9246b99e8887f03180d0c3" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:24.670656 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc28b2028a9246b99e8887f03180d0c3" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:24.670756 30407 raft_consensus.cc:3037] T fc28b2028a9246b99e8887f03180d0c3 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.670765 30276 raft_consensus.cc:3037] T fc28b2028a9246b99e8887f03180d0c3 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.671553 30407 raft_consensus.cc:2454] T fc28b2028a9246b99e8887f03180d0c3 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:24.671617 30276 raft_consensus.cc:2454] T fc28b2028a9246b99e8887f03180d0c3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:24.671826 30081 leader_election.cc:302] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:24.671973 31061 raft_consensus.cc:2781] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:24.672041 31061 raft_consensus.cc:686] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:24.672200 31061 consensus_queue.cc:237] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:24.673892 29865 catalog_manager.cc:5484] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:24.751689 31449 raft_consensus.cc:482] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:24.751794 31449 raft_consensus.cc:504] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:24.752110 31449 leader_election.cc:288] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:24.752453 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d406c64b2e432a88e8c675b91dd144" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:24.752465 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d406c64b2e432a88e8c675b91dd144" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:24.752573 30407 raft_consensus.cc:2454] T c5d406c64b2e432a88e8c675b91dd144 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:24.752589 30013 raft_consensus.cc:2454] T c5d406c64b2e432a88e8c675b91dd144 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:24.752821 30210 leader_election.cc:302] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:24.752974 31449 raft_consensus.cc:2781] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:24.753026 31449 raft_consensus.cc:482] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:24.753051 31449 raft_consensus.cc:3037] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.753821 31449 raft_consensus.cc:504] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:24.754122 31449 leader_election.cc:288] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:24.754395 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d406c64b2e432a88e8c675b91dd144" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:24.754462 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d406c64b2e432a88e8c675b91dd144" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:24.754489 30407 raft_consensus.cc:3037] T c5d406c64b2e432a88e8c675b91dd144 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.754567 30013 raft_consensus.cc:3037] T c5d406c64b2e432a88e8c675b91dd144 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.755198 30407 raft_consensus.cc:2454] T c5d406c64b2e432a88e8c675b91dd144 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:24.755348 30013 raft_consensus.cc:2454] T c5d406c64b2e432a88e8c675b91dd144 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:24.755450 30210 leader_election.cc:302] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:24.755618 31449 raft_consensus.cc:2781] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:24.755807 31449 raft_consensus.cc:686] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:24.755935 31449 consensus_queue.cc:237] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:24.757470 29865 catalog_manager.cc:5484] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:24.890136 31061 raft_consensus.cc:482] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:24.890249 31061 raft_consensus.cc:504] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:24.890589 31061 leader_election.cc:288] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:24.890885 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d87b2a4f6764e20afbfb43681f58467" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:24.890970 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d87b2a4f6764e20afbfb43681f58467" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:24.891021 30277 raft_consensus.cc:2454] T 8d87b2a4f6764e20afbfb43681f58467 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:24.891120 30013 raft_consensus.cc:2454] T 8d87b2a4f6764e20afbfb43681f58467 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:24.891309 30081 leader_election.cc:302] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:24.891469 31061 raft_consensus.cc:2781] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:24.891523 31061 raft_consensus.cc:482] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:24.891552 31061 raft_consensus.cc:3037] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.892321 31061 raft_consensus.cc:504] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:24.892593 31061 leader_election.cc:288] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:24.892844 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d87b2a4f6764e20afbfb43681f58467" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:24.892916 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d87b2a4f6764e20afbfb43681f58467" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:24.892958 30277 raft_consensus.cc:3037] T 8d87b2a4f6764e20afbfb43681f58467 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.893015 30013 raft_consensus.cc:3037] T 8d87b2a4f6764e20afbfb43681f58467 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.893868 30013 raft_consensus.cc:2454] T 8d87b2a4f6764e20afbfb43681f58467 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:24.893867 30277 raft_consensus.cc:2454] T 8d87b2a4f6764e20afbfb43681f58467 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:24.894181 30081 leader_election.cc:302] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:24.894343 31061 raft_consensus.cc:2781] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:24.894429 31061 raft_consensus.cc:686] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:24.894564 31061 consensus_queue.cc:237] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:24.896346 29865 catalog_manager.cc:5484] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
W20240501 14:09:25.015630 31544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:25.015826 31544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:25.018957 31544 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 14:09:25.094447 30407 raft_consensus.cc:1261] T c5d406c64b2e432a88e8c675b91dd144 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:25.094722 30013 raft_consensus.cc:1261] T c5d406c64b2e432a88e8c675b91dd144 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:25.094883 31541 consensus_queue.cc:1035] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:25.095217 31449 consensus_queue.cc:1035] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:25.100313 31575 mvcc.cc:205] Tried to move back new op lower bound from 7022889226623070208 to 7022889225241587712. Current Snapshot: MvccSnapshot[applied={T|T < 7022889226623070208}]
I20240501 14:09:25.106763 30277 raft_consensus.cc:1261] T 8d87b2a4f6764e20afbfb43681f58467 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:25.107271 31061 consensus_queue.cc:1035] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:25.108021 30013 raft_consensus.cc:1261] T 8d87b2a4f6764e20afbfb43681f58467 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:25.108968 31061 consensus_queue.cc:1035] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:25.117120 30407 raft_consensus.cc:1261] T fc28b2028a9246b99e8887f03180d0c3 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:25.117625 31061 consensus_queue.cc:1035] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:25.121016 30277 raft_consensus.cc:1261] T fc28b2028a9246b99e8887f03180d0c3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:25.121429 31440 consensus_queue.cc:1035] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:25.132154 31577 mvcc.cc:205] Tried to move back new op lower bound from 7022889226685972480 to 7022889224898625536. Current Snapshot: MvccSnapshot[applied={T|T < 7022889226685972480}]
INSERT report
rows total: 2000
time total: 101.303 ms
time per row: 0.0506515 ms
I20240501 14:09:25.160048 29866 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:25.164724 29994 tablet_service.cc:1467] Processing CreateTablet for tablet fb466c8cb5a74897a884ac6012cc3734 (DEFAULT_TABLE table=rebalance_test_table_8 [id=40bc333ce8c9447da3fd22a75d02001d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.164786 30126 tablet_service.cc:1467] Processing CreateTablet for tablet fb466c8cb5a74897a884ac6012cc3734 (DEFAULT_TABLE table=rebalance_test_table_8 [id=40bc333ce8c9447da3fd22a75d02001d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.165030 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb466c8cb5a74897a884ac6012cc3734. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.165041 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb466c8cb5a74897a884ac6012cc3734. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.165709 30256 tablet_service.cc:1467] Processing CreateTablet for tablet fb466c8cb5a74897a884ac6012cc3734 (DEFAULT_TABLE table=rebalance_test_table_8 [id=40bc333ce8c9447da3fd22a75d02001d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.165971 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb466c8cb5a74897a884ac6012cc3734. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.165992 30125 tablet_service.cc:1467] Processing CreateTablet for tablet 3c9856cd2c16497aa40bd041d5e0a297 (DEFAULT_TABLE table=rebalance_test_table_8 [id=40bc333ce8c9447da3fd22a75d02001d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.166226 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c9856cd2c16497aa40bd041d5e0a297. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.167285 31598 tablet_bootstrap.cc:492] T fb466c8cb5a74897a884ac6012cc3734 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:25.167272 30255 tablet_service.cc:1467] Processing CreateTablet for tablet ad69f22c52c34610bd47091f7305e19a (DEFAULT_TABLE table=rebalance_test_table_8 [id=40bc333ce8c9447da3fd22a75d02001d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.167503 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad69f22c52c34610bd47091f7305e19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.168180 31598 tablet_bootstrap.cc:654] T fb466c8cb5a74897a884ac6012cc3734 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.168998 29994 tablet_service.cc:1467] Processing CreateTablet for tablet 3c9856cd2c16497aa40bd041d5e0a297 (DEFAULT_TABLE table=rebalance_test_table_8 [id=40bc333ce8c9447da3fd22a75d02001d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.169222 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c9856cd2c16497aa40bd041d5e0a297. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.169585 30387 tablet_service.cc:1467] Processing CreateTablet for tablet ad69f22c52c34610bd47091f7305e19a (DEFAULT_TABLE table=rebalance_test_table_8 [id=40bc333ce8c9447da3fd22a75d02001d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.169839 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad69f22c52c34610bd47091f7305e19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.171123 29993 tablet_service.cc:1467] Processing CreateTablet for tablet ad69f22c52c34610bd47091f7305e19a (DEFAULT_TABLE table=rebalance_test_table_8 [id=40bc333ce8c9447da3fd22a75d02001d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.171339 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad69f22c52c34610bd47091f7305e19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.172020 31601 tablet_bootstrap.cc:492] T ad69f22c52c34610bd47091f7305e19a P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:25.172108 31597 tablet_bootstrap.cc:492] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:25.172397 31599 tablet_bootstrap.cc:492] T fb466c8cb5a74897a884ac6012cc3734 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:25.172559 30386 tablet_service.cc:1467] Processing CreateTablet for tablet 3c9856cd2c16497aa40bd041d5e0a297 (DEFAULT_TABLE table=rebalance_test_table_8 [id=40bc333ce8c9447da3fd22a75d02001d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.172775 30386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c9856cd2c16497aa40bd041d5e0a297. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.173017 31597 tablet_bootstrap.cc:654] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.173168 31601 tablet_bootstrap.cc:654] T ad69f22c52c34610bd47091f7305e19a P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.173265 31599 tablet_bootstrap.cc:654] T fb466c8cb5a74897a884ac6012cc3734 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.174268 31601 tablet_bootstrap.cc:492] T ad69f22c52c34610bd47091f7305e19a P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:25.174347 31601 ts_tablet_manager.cc:1367] T ad69f22c52c34610bd47091f7305e19a P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:25.174721 31601 raft_consensus.cc:348] T ad69f22c52c34610bd47091f7305e19a P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:25.174791 31601 raft_consensus.cc:374] T ad69f22c52c34610bd47091f7305e19a P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.174810 31601 raft_consensus.cc:729] T ad69f22c52c34610bd47091f7305e19a P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.174921 31601 consensus_queue.cc:260] T ad69f22c52c34610bd47091f7305e19a P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:25.175190 31601 ts_tablet_manager.cc:1398] T ad69f22c52c34610bd47091f7305e19a P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:25.175330 31601 tablet_bootstrap.cc:492] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:25.175354 31598 tablet_bootstrap.cc:492] T fb466c8cb5a74897a884ac6012cc3734 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:25.175408 31598 ts_tablet_manager.cc:1367] T fb466c8cb5a74897a884ac6012cc3734 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:25.175737 31598 raft_consensus.cc:348] T fb466c8cb5a74897a884ac6012cc3734 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:25.175830 31598 raft_consensus.cc:374] T fb466c8cb5a74897a884ac6012cc3734 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.175865 31598 raft_consensus.cc:729] T fb466c8cb5a74897a884ac6012cc3734 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.175989 31598 consensus_queue.cc:260] T fb466c8cb5a74897a884ac6012cc3734 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:25.176100 31601 tablet_bootstrap.cc:654] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.176255 31598 ts_tablet_manager.cc:1398] T fb466c8cb5a74897a884ac6012cc3734 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:25.176396 31598 tablet_bootstrap.cc:492] T 3c9856cd2c16497aa40bd041d5e0a297 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:25.177069 31598 tablet_bootstrap.cc:654] T 3c9856cd2c16497aa40bd041d5e0a297 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.177659 31601 tablet_bootstrap.cc:492] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:25.177732 31601 ts_tablet_manager.cc:1367] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:25.177910 31599 tablet_bootstrap.cc:492] T fb466c8cb5a74897a884ac6012cc3734 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:25.177979 31599 ts_tablet_manager.cc:1367] T fb466c8cb5a74897a884ac6012cc3734 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:25.178097 31597 tablet_bootstrap.cc:492] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:25.178192 31597 ts_tablet_manager.cc:1367] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:25.178162 31601 raft_consensus.cc:348] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:25.178344 31601 raft_consensus.cc:374] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.178393 31601 raft_consensus.cc:729] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.178505 31601 consensus_queue.cc:260] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:25.178577 31599 raft_consensus.cc:348] T fb466c8cb5a74897a884ac6012cc3734 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:25.178666 31599 raft_consensus.cc:374] T fb466c8cb5a74897a884ac6012cc3734 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.178696 31599 raft_consensus.cc:729] T fb466c8cb5a74897a884ac6012cc3734 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.178782 31601 ts_tablet_manager.cc:1398] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:25.178738 31597 raft_consensus.cc:348] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:25.178843 31597 raft_consensus.cc:374] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.178814 31599 consensus_queue.cc:260] T fb466c8cb5a74897a884ac6012cc3734 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:25.178879 31597 raft_consensus.cc:729] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.179067 31597 consensus_queue.cc:260] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:25.179389 31597 ts_tablet_manager.cc:1398] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:25.179594 31599 ts_tablet_manager.cc:1398] T fb466c8cb5a74897a884ac6012cc3734 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:25.179602 31597 tablet_bootstrap.cc:492] T 3c9856cd2c16497aa40bd041d5e0a297 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:25.179754 31599 tablet_bootstrap.cc:492] T ad69f22c52c34610bd47091f7305e19a P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:25.180653 31597 tablet_bootstrap.cc:654] T 3c9856cd2c16497aa40bd041d5e0a297 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.180686 31599 tablet_bootstrap.cc:654] T ad69f22c52c34610bd47091f7305e19a P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.181640 31597 tablet_bootstrap.cc:492] T 3c9856cd2c16497aa40bd041d5e0a297 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:25.181672 31598 tablet_bootstrap.cc:492] T 3c9856cd2c16497aa40bd041d5e0a297 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:25.181708 31597 ts_tablet_manager.cc:1367] T 3c9856cd2c16497aa40bd041d5e0a297 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:25.181749 31598 ts_tablet_manager.cc:1367] T 3c9856cd2c16497aa40bd041d5e0a297 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:25.182123 31598 raft_consensus.cc:348] T 3c9856cd2c16497aa40bd041d5e0a297 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:25.182220 31598 raft_consensus.cc:374] T 3c9856cd2c16497aa40bd041d5e0a297 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.182101 31597 raft_consensus.cc:348] T 3c9856cd2c16497aa40bd041d5e0a297 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:25.182250 31598 raft_consensus.cc:729] T 3c9856cd2c16497aa40bd041d5e0a297 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.182265 31597 raft_consensus.cc:374] T 3c9856cd2c16497aa40bd041d5e0a297 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.182308 31597 raft_consensus.cc:729] T 3c9856cd2c16497aa40bd041d5e0a297 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.182356 31598 consensus_queue.cc:260] T 3c9856cd2c16497aa40bd041d5e0a297 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:25.182418 31597 consensus_queue.cc:260] T 3c9856cd2c16497aa40bd041d5e0a297 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:25.182596 31598 ts_tablet_manager.cc:1398] T 3c9856cd2c16497aa40bd041d5e0a297 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:25.182627 31597 ts_tablet_manager.cc:1398] T 3c9856cd2c16497aa40bd041d5e0a297 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:25.182744 31598 tablet_bootstrap.cc:492] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:25.183563 31598 tablet_bootstrap.cc:654] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.184367 31061 raft_consensus.cc:482] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:25.184453 31061 raft_consensus.cc:504] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:25.184495 31598 tablet_bootstrap.cc:492] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:25.184567 31598 ts_tablet_manager.cc:1367] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:25.184787 31061 leader_election.cc:288] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:25.185077 31598 raft_consensus.cc:348] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:25.185166 31598 raft_consensus.cc:374] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.185195 31598 raft_consensus.cc:729] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.185338 31598 consensus_queue.cc:260] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:25.185614 31598 ts_tablet_manager.cc:1398] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:25.185705 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb466c8cb5a74897a884ac6012cc3734" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:25.185757 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb466c8cb5a74897a884ac6012cc3734" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:25.185791 30277 raft_consensus.cc:2454] T fb466c8cb5a74897a884ac6012cc3734 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:25.185834 30013 raft_consensus.cc:2454] T fb466c8cb5a74897a884ac6012cc3734 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:25.186059 30081 leader_election.cc:302] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:25.186329 31061 raft_consensus.cc:2781] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:25.186385 31061 raft_consensus.cc:482] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:25.186422 31061 raft_consensus.cc:3037] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.187419 31061 raft_consensus.cc:504] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:25.187777 31061 leader_election.cc:288] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:25.188333 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb466c8cb5a74897a884ac6012cc3734" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:25.188334 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb466c8cb5a74897a884ac6012cc3734" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:25.188427 30277 raft_consensus.cc:3037] T fb466c8cb5a74897a884ac6012cc3734 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.188432 30013 raft_consensus.cc:3037] T fb466c8cb5a74897a884ac6012cc3734 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.189133 30013 raft_consensus.cc:2454] T fb466c8cb5a74897a884ac6012cc3734 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:25.189334 30277 raft_consensus.cc:2454] T fb466c8cb5a74897a884ac6012cc3734 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:25.189419 30081 leader_election.cc:302] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:25.189790 31061 raft_consensus.cc:2781] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:25.189886 31061 raft_consensus.cc:686] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:25.190021 31061 consensus_queue.cc:237] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:25.190408 31238 raft_consensus.cc:482] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:25.190495 31238 raft_consensus.cc:504] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:25.190814 31238 leader_election.cc:288] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:25.191172 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9856cd2c16497aa40bd041d5e0a297" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:25.191322 30013 raft_consensus.cc:2454] T 3c9856cd2c16497aa40bd041d5e0a297 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:25.191500 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9856cd2c16497aa40bd041d5e0a297" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:25.191525 30343 leader_election.cc:302] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:25.191592 30145 raft_consensus.cc:2454] T 3c9856cd2c16497aa40bd041d5e0a297 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:25.191939 29866 catalog_manager.cc:5484] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:25.192495 31238 raft_consensus.cc:2781] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:25.192562 31238 raft_consensus.cc:482] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:25.192600 31238 raft_consensus.cc:3037] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.193595 31238 raft_consensus.cc:504] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:25.193950 31238 leader_election.cc:288] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:25.194267 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9856cd2c16497aa40bd041d5e0a297" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:25.194377 30145 raft_consensus.cc:3037] T 3c9856cd2c16497aa40bd041d5e0a297 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.194370 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9856cd2c16497aa40bd041d5e0a297" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:25.194459 30013 raft_consensus.cc:3037] T 3c9856cd2c16497aa40bd041d5e0a297 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.195051 30145 raft_consensus.cc:2454] T 3c9856cd2c16497aa40bd041d5e0a297 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:25.195186 30013 raft_consensus.cc:2454] T 3c9856cd2c16497aa40bd041d5e0a297 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:25.195470 30343 leader_election.cc:302] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:25.195677 31238 raft_consensus.cc:2781] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:25.195860 31238 raft_consensus.cc:686] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:25.196007 31238 consensus_queue.cc:237] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:25.197885 29866 catalog_manager.cc:5484] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:25.201002 31599 tablet_bootstrap.cc:492] T ad69f22c52c34610bd47091f7305e19a P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:25.201104 31599 ts_tablet_manager.cc:1367] T ad69f22c52c34610bd47091f7305e19a P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.001s
I20240501 14:09:25.201512 31599 raft_consensus.cc:348] T ad69f22c52c34610bd47091f7305e19a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:25.201602 31599 raft_consensus.cc:374] T ad69f22c52c34610bd47091f7305e19a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.201637 31599 raft_consensus.cc:729] T ad69f22c52c34610bd47091f7305e19a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.201761 31599 consensus_queue.cc:260] T ad69f22c52c34610bd47091f7305e19a P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:25.202039 31599 ts_tablet_manager.cc:1398] T ad69f22c52c34610bd47091f7305e19a P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:25.429210 31019 raft_consensus.cc:482] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:25.429325 31019 raft_consensus.cc:504] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:25.429668 31019 leader_election.cc:288] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:25.429960 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad69f22c52c34610bd47091f7305e19a" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:25.429955 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad69f22c52c34610bd47091f7305e19a" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:25.430088 30407 raft_consensus.cc:2454] T ad69f22c52c34610bd47091f7305e19a P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:25.430099 30277 raft_consensus.cc:2454] T ad69f22c52c34610bd47091f7305e19a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:25.430384 29950 leader_election.cc:302] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:25.430564 31019 raft_consensus.cc:2781] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:25.430619 31019 raft_consensus.cc:482] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:25.430667 31019 raft_consensus.cc:3037] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.431463 31019 raft_consensus.cc:504] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:25.431754 31019 leader_election.cc:288] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:25.431980 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad69f22c52c34610bd47091f7305e19a" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:25.432044 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad69f22c52c34610bd47091f7305e19a" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:25.432101 30407 raft_consensus.cc:3037] T ad69f22c52c34610bd47091f7305e19a P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.432173 30277 raft_consensus.cc:3037] T ad69f22c52c34610bd47091f7305e19a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.432909 30277 raft_consensus.cc:2454] T ad69f22c52c34610bd47091f7305e19a P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:25.433018 30407 raft_consensus.cc:2454] T ad69f22c52c34610bd47091f7305e19a P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:25.433178 29950 leader_election.cc:302] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:25.433352 31019 raft_consensus.cc:2781] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:25.433547 31019 raft_consensus.cc:686] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:25.433701 31019 consensus_queue.cc:237] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:25.436110 29866 catalog_manager.cc:5484] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
W20240501 14:09:25.564258 31615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:25.564442 31615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:25.566988 31615 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 14:09:25.640465 30145 raft_consensus.cc:1261] T 3c9856cd2c16497aa40bd041d5e0a297 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:25.640643 30013 raft_consensus.cc:1261] T 3c9856cd2c16497aa40bd041d5e0a297 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:25.641301 31611 consensus_queue.cc:1035] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:25.641559 31238 consensus_queue.cc:1035] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:25.644524 30277 raft_consensus.cc:1261] T ad69f22c52c34610bd47091f7305e19a P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:25.644901 31019 consensus_queue.cc:1035] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:25.650077 30014 raft_consensus.cc:1261] T fb466c8cb5a74897a884ac6012cc3734 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:25.650312 30276 raft_consensus.cc:1261] T fb466c8cb5a74897a884ac6012cc3734 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:25.650429 31061 consensus_queue.cc:1035] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:25.650677 31591 consensus_queue.cc:1035] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:25.651062 30407 raft_consensus.cc:1261] T ad69f22c52c34610bd47091f7305e19a P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:25.651770 31019 consensus_queue.cc:1035] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 102.881 ms
time per row: 0.0514403 ms
I20240501 14:09:25.711122 29869 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:25.716058 30125 tablet_service.cc:1467] Processing CreateTablet for tablet 940c8f54339f4a32be2942682a41c685 (DEFAULT_TABLE table=rebalance_test_table_9 [id=abd0f09508ae4b6699a526d040fd6f96]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.716167 30255 tablet_service.cc:1467] Processing CreateTablet for tablet 940c8f54339f4a32be2942682a41c685 (DEFAULT_TABLE table=rebalance_test_table_9 [id=abd0f09508ae4b6699a526d040fd6f96]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.716434 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940c8f54339f4a32be2942682a41c685. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.716451 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940c8f54339f4a32be2942682a41c685. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.717036 30256 tablet_service.cc:1467] Processing CreateTablet for tablet 3125c67fe40641c2bfd5dcac81e7237f (DEFAULT_TABLE table=rebalance_test_table_9 [id=abd0f09508ae4b6699a526d040fd6f96]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.717056 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 02640dad4f4a4285a9e1380bf108b791 (DEFAULT_TABLE table=rebalance_test_table_9 [id=abd0f09508ae4b6699a526d040fd6f96]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.717284 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3125c67fe40641c2bfd5dcac81e7237f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.717298 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02640dad4f4a4285a9e1380bf108b791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.719125 30126 tablet_service.cc:1467] Processing CreateTablet for tablet 02640dad4f4a4285a9e1380bf108b791 (DEFAULT_TABLE table=rebalance_test_table_9 [id=abd0f09508ae4b6699a526d040fd6f96]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.719218 31677 tablet_bootstrap.cc:492] T 940c8f54339f4a32be2942682a41c685 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:25.719341 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02640dad4f4a4285a9e1380bf108b791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.719364 30257 tablet_service.cc:1467] Processing CreateTablet for tablet 02640dad4f4a4285a9e1380bf108b791 (DEFAULT_TABLE table=rebalance_test_table_9 [id=abd0f09508ae4b6699a526d040fd6f96]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.719595 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02640dad4f4a4285a9e1380bf108b791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.720093 31678 tablet_bootstrap.cc:492] T 02640dad4f4a4285a9e1380bf108b791 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:25.720134 31677 tablet_bootstrap.cc:654] T 940c8f54339f4a32be2942682a41c685 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.720808 31676 tablet_bootstrap.cc:492] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:25.721140 31678 tablet_bootstrap.cc:654] T 02640dad4f4a4285a9e1380bf108b791 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.721680 31676 tablet_bootstrap.cc:654] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.721843 30386 tablet_service.cc:1467] Processing CreateTablet for tablet 940c8f54339f4a32be2942682a41c685 (DEFAULT_TABLE table=rebalance_test_table_9 [id=abd0f09508ae4b6699a526d040fd6f96]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.722122 30386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940c8f54339f4a32be2942682a41c685. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.722225 30387 tablet_service.cc:1467] Processing CreateTablet for tablet 3125c67fe40641c2bfd5dcac81e7237f (DEFAULT_TABLE table=rebalance_test_table_9 [id=abd0f09508ae4b6699a526d040fd6f96]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.722298 31678 tablet_bootstrap.cc:492] T 02640dad4f4a4285a9e1380bf108b791 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:25.722368 31678 ts_tablet_manager.cc:1367] T 02640dad4f4a4285a9e1380bf108b791 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:25.722445 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3125c67fe40641c2bfd5dcac81e7237f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.722791 31678 raft_consensus.cc:348] T 02640dad4f4a4285a9e1380bf108b791 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:25.722887 31678 raft_consensus.cc:374] T 02640dad4f4a4285a9e1380bf108b791 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.722913 31678 raft_consensus.cc:729] T 02640dad4f4a4285a9e1380bf108b791 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.723031 31678 consensus_queue.cc:260] T 02640dad4f4a4285a9e1380bf108b791 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:25.723297 31678 ts_tablet_manager.cc:1398] T 02640dad4f4a4285a9e1380bf108b791 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:25.723640 31682 tablet_bootstrap.cc:492] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:25.724367 31676 tablet_bootstrap.cc:492] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:25.724453 31676 ts_tablet_manager.cc:1367] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:25.724792 30125 tablet_service.cc:1467] Processing CreateTablet for tablet 3125c67fe40641c2bfd5dcac81e7237f (DEFAULT_TABLE table=rebalance_test_table_9 [id=abd0f09508ae4b6699a526d040fd6f96]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.724886 31676 raft_consensus.cc:348] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:25.725028 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3125c67fe40641c2bfd5dcac81e7237f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.725585 31677 tablet_bootstrap.cc:492] T 940c8f54339f4a32be2942682a41c685 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:25.725663 31677 ts_tablet_manager.cc:1367] T 940c8f54339f4a32be2942682a41c685 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:25.725951 31682 tablet_bootstrap.cc:654] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.726100 31677 raft_consensus.cc:348] T 940c8f54339f4a32be2942682a41c685 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:25.726204 31677 raft_consensus.cc:374] T 940c8f54339f4a32be2942682a41c685 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.726234 31677 raft_consensus.cc:729] T 940c8f54339f4a32be2942682a41c685 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.726338 31677 consensus_queue.cc:260] T 940c8f54339f4a32be2942682a41c685 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:25.726615 31677 ts_tablet_manager.cc:1398] T 940c8f54339f4a32be2942682a41c685 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:25.726760 31677 tablet_bootstrap.cc:492] T 3125c67fe40641c2bfd5dcac81e7237f P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:25.725019 31676 raft_consensus.cc:374] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.726836 31676 raft_consensus.cc:729] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.726953 31682 tablet_bootstrap.cc:492] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:25.726979 31676 consensus_queue.cc:260] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:25.727035 31682 ts_tablet_manager.cc:1367] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:25.727330 31676 ts_tablet_manager.cc:1398] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:25.727460 31676 tablet_bootstrap.cc:492] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:25.727454 31682 raft_consensus.cc:348] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:25.727524 31682 raft_consensus.cc:374] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.727545 31682 raft_consensus.cc:729] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.727584 31677 tablet_bootstrap.cc:654] T 3125c67fe40641c2bfd5dcac81e7237f P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.727639 31682 consensus_queue.cc:260] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:25.727869 31682 ts_tablet_manager.cc:1398] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:25.727974 31682 tablet_bootstrap.cc:492] T 3125c67fe40641c2bfd5dcac81e7237f P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:25.728334 31676 tablet_bootstrap.cc:654] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.728883 31682 tablet_bootstrap.cc:654] T 3125c67fe40641c2bfd5dcac81e7237f P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.729265 31676 tablet_bootstrap.cc:492] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:25.729337 31676 ts_tablet_manager.cc:1367] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:25.729736 31677 tablet_bootstrap.cc:492] T 3125c67fe40641c2bfd5dcac81e7237f P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:25.729740 31676 raft_consensus.cc:348] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:25.729813 31677 ts_tablet_manager.cc:1367] T 3125c67fe40641c2bfd5dcac81e7237f P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:25.729821 31676 raft_consensus.cc:374] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.729852 31676 raft_consensus.cc:729] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.729969 31676 consensus_queue.cc:260] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:25.730235 31676 ts_tablet_manager.cc:1398] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:25.730235 31677 raft_consensus.cc:348] T 3125c67fe40641c2bfd5dcac81e7237f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:25.730320 31677 raft_consensus.cc:374] T 3125c67fe40641c2bfd5dcac81e7237f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.730353 31677 raft_consensus.cc:729] T 3125c67fe40641c2bfd5dcac81e7237f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.730379 31676 tablet_bootstrap.cc:492] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:25.730813 31677 consensus_queue.cc:260] T 3125c67fe40641c2bfd5dcac81e7237f P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:25.731036 31682 tablet_bootstrap.cc:492] T 3125c67fe40641c2bfd5dcac81e7237f P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:25.731125 31682 ts_tablet_manager.cc:1367] T 3125c67fe40641c2bfd5dcac81e7237f P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:25.731194 31676 tablet_bootstrap.cc:654] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.731825 31677 ts_tablet_manager.cc:1398] T 3125c67fe40641c2bfd5dcac81e7237f P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:25.732120 31677 tablet_bootstrap.cc:492] T 02640dad4f4a4285a9e1380bf108b791 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:25.732076 31682 raft_consensus.cc:348] T 3125c67fe40641c2bfd5dcac81e7237f P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:25.732174 31682 raft_consensus.cc:374] T 3125c67fe40641c2bfd5dcac81e7237f P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.732215 31682 raft_consensus.cc:729] T 3125c67fe40641c2bfd5dcac81e7237f P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.732419 31676 tablet_bootstrap.cc:492] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:25.732499 31676 ts_tablet_manager.cc:1367] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:09:25.732650 31682 consensus_queue.cc:260] T 3125c67fe40641c2bfd5dcac81e7237f P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:25.732890 31676 raft_consensus.cc:348] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:25.733003 31676 raft_consensus.cc:374] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.733042 31676 raft_consensus.cc:729] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.733137 31676 consensus_queue.cc:260] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:25.733171 31677 tablet_bootstrap.cc:654] T 02640dad4f4a4285a9e1380bf108b791 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.733383 31676 ts_tablet_manager.cc:1398] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:25.733974 31682 ts_tablet_manager.cc:1398] T 3125c67fe40641c2bfd5dcac81e7237f P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:25.734691 31677 tablet_bootstrap.cc:492] T 02640dad4f4a4285a9e1380bf108b791 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:25.734769 31677 ts_tablet_manager.cc:1367] T 02640dad4f4a4285a9e1380bf108b791 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:25.735321 31677 raft_consensus.cc:348] T 02640dad4f4a4285a9e1380bf108b791 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:25.735425 31677 raft_consensus.cc:374] T 02640dad4f4a4285a9e1380bf108b791 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.735456 31677 raft_consensus.cc:729] T 02640dad4f4a4285a9e1380bf108b791 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.735582 31677 consensus_queue.cc:260] T 02640dad4f4a4285a9e1380bf108b791 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:25.735874 31677 ts_tablet_manager.cc:1398] T 02640dad4f4a4285a9e1380bf108b791 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:25.756064 31238 raft_consensus.cc:482] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:25.756165 31238 raft_consensus.cc:504] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:25.756572 31238 leader_election.cc:288] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:25.756858 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c8f54339f4a32be2942682a41c685" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:25.756908 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c8f54339f4a32be2942682a41c685" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:25.756963 30145 raft_consensus.cc:2454] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:25.757046 30276 raft_consensus.cc:2454] T 940c8f54339f4a32be2942682a41c685 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:25.757248 30344 leader_election.cc:302] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:25.757393 31238 raft_consensus.cc:2781] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:25.757448 31238 raft_consensus.cc:482] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:25.757483 31238 raft_consensus.cc:3037] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.758149 31238 raft_consensus.cc:504] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:25.758456 31238 leader_election.cc:288] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:25.758708 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c8f54339f4a32be2942682a41c685" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:25.758749 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c8f54339f4a32be2942682a41c685" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:25.758805 30145 raft_consensus.cc:3037] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.758836 30276 raft_consensus.cc:3037] T 940c8f54339f4a32be2942682a41c685 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.759471 30145 raft_consensus.cc:2454] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:25.759526 30276 raft_consensus.cc:2454] T 940c8f54339f4a32be2942682a41c685 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:25.759755 30344 leader_election.cc:302] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:25.759912 31238 raft_consensus.cc:2781] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:25.759997 31238 raft_consensus.cc:686] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:25.760108 31238 consensus_queue.cc:237] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:25.761678 29869 catalog_manager.cc:5484] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:25.813365 31591 raft_consensus.cc:482] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:25.813480 31591 raft_consensus.cc:504] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:25.813819 31591 leader_election.cc:288] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:25.814111 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3125c67fe40641c2bfd5dcac81e7237f" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:25.814224 30277 raft_consensus.cc:2454] T 3125c67fe40641c2bfd5dcac81e7237f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:25.814170 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3125c67fe40641c2bfd5dcac81e7237f" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:25.814322 30407 raft_consensus.cc:2454] T 3125c67fe40641c2bfd5dcac81e7237f P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:25.814491 30081 leader_election.cc:302] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:25.814656 31591 raft_consensus.cc:2781] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:25.814724 31591 raft_consensus.cc:482] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:25.814772 31591 raft_consensus.cc:3037] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.815536 31591 raft_consensus.cc:504] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:25.815800 31591 leader_election.cc:288] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:25.816066 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3125c67fe40641c2bfd5dcac81e7237f" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:25.816066 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3125c67fe40641c2bfd5dcac81e7237f" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:25.816184 30277 raft_consensus.cc:3037] T 3125c67fe40641c2bfd5dcac81e7237f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.816193 30407 raft_consensus.cc:3037] T 3125c67fe40641c2bfd5dcac81e7237f P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.817075 30407 raft_consensus.cc:2454] T 3125c67fe40641c2bfd5dcac81e7237f P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:25.817075 30277 raft_consensus.cc:2454] T 3125c67fe40641c2bfd5dcac81e7237f P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:25.817333 30079 leader_election.cc:302] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:25.817488 31591 raft_consensus.cc:2781] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:25.817567 31591 raft_consensus.cc:686] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:25.817711 31591 consensus_queue.cc:237] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:25.819554 29869 catalog_manager.cc:5484] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:25.912622 31591 raft_consensus.cc:482] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:25.912762 31591 raft_consensus.cc:504] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:25.913189 31591 leader_election.cc:288] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:25.913589 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02640dad4f4a4285a9e1380bf108b791" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:25.913627 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02640dad4f4a4285a9e1380bf108b791" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:25.913762 30014 raft_consensus.cc:2454] T 02640dad4f4a4285a9e1380bf108b791 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:25.913753 30277 raft_consensus.cc:2454] T 02640dad4f4a4285a9e1380bf108b791 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:25.914031 30081 leader_election.cc:302] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:25.914203 31591 raft_consensus.cc:2781] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:25.914261 31591 raft_consensus.cc:482] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:25.914309 31591 raft_consensus.cc:3037] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.915099 31591 raft_consensus.cc:504] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:25.915396 31591 leader_election.cc:288] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:25.915688 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02640dad4f4a4285a9e1380bf108b791" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:25.915787 30277 raft_consensus.cc:3037] T 02640dad4f4a4285a9e1380bf108b791 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.915779 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02640dad4f4a4285a9e1380bf108b791" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:25.915899 30014 raft_consensus.cc:3037] T 02640dad4f4a4285a9e1380bf108b791 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.916646 30014 raft_consensus.cc:2454] T 02640dad4f4a4285a9e1380bf108b791 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:25.916685 30277 raft_consensus.cc:2454] T 02640dad4f4a4285a9e1380bf108b791 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:25.916920 30081 leader_election.cc:302] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:25.917094 31591 raft_consensus.cc:2781] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:25.917179 31591 raft_consensus.cc:686] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:25.917351 31591 consensus_queue.cc:237] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:25.919142 29869 catalog_manager.cc:5484] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
W20240501 14:09:26.063458 31692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:26.063683 31692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:26.066605 31692 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 14:09:26.146088 30407 raft_consensus.cc:1261] T 3125c67fe40641c2bfd5dcac81e7237f P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.146448 30277 raft_consensus.cc:1261] T 3125c67fe40641c2bfd5dcac81e7237f P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.146448 30014 raft_consensus.cc:1261] T 02640dad4f4a4285a9e1380bf108b791 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.146838 31061 consensus_queue.cc:1035] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:26.146989 31591 consensus_queue.cc:1035] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:26.147173 31061 consensus_queue.cc:1035] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:26.150663 30277 raft_consensus.cc:1261] T 940c8f54339f4a32be2942682a41c685 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.151010 31238 consensus_queue.cc:1035] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:26.151502 30145 raft_consensus.cc:1261] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.151885 31238 consensus_queue.cc:1035] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:26.153302 30264 raft_consensus.cc:1261] T 02640dad4f4a4285a9e1380bf108b791 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.158071 31061 consensus_queue.cc:1035] T 02640dad4f4a4285a9e1380bf108b791 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 99.6011 ms
time per row: 0.0498005 ms
I20240501 14:09:26.215440 29867 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:26.220672 30125 tablet_service.cc:1467] Processing CreateTablet for tablet 62b3902e073b42b18c3ff62bcdccb2cb (DEFAULT_TABLE table=rebalance_test_table_10 [id=4fa8cc9c35c649efb9c70f46be41dd50]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.220782 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 62b3902e073b42b18c3ff62bcdccb2cb (DEFAULT_TABLE table=rebalance_test_table_10 [id=4fa8cc9c35c649efb9c70f46be41dd50]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.220837 30387 tablet_service.cc:1467] Processing CreateTablet for tablet 62b3902e073b42b18c3ff62bcdccb2cb (DEFAULT_TABLE table=rebalance_test_table_10 [id=4fa8cc9c35c649efb9c70f46be41dd50]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.220965 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62b3902e073b42b18c3ff62bcdccb2cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.221074 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62b3902e073b42b18c3ff62bcdccb2cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.221074 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62b3902e073b42b18c3ff62bcdccb2cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.222756 31676 tablet_bootstrap.cc:492] T 62b3902e073b42b18c3ff62bcdccb2cb P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:26.223063 31678 tablet_bootstrap.cc:492] T 62b3902e073b42b18c3ff62bcdccb2cb P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:26.223551 31676 tablet_bootstrap.cc:654] T 62b3902e073b42b18c3ff62bcdccb2cb P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.223591 30257 tablet_service.cc:1467] Processing CreateTablet for tablet 9f15d8d22c324ff3a3315ecb2345553d (DEFAULT_TABLE table=rebalance_test_table_10 [id=4fa8cc9c35c649efb9c70f46be41dd50]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.223654 29994 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5e1afe21654080b5aa3eb877e8f1d9 (DEFAULT_TABLE table=rebalance_test_table_10 [id=4fa8cc9c35c649efb9c70f46be41dd50]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.223834 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5e1afe21654080b5aa3eb877e8f1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.223836 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f15d8d22c324ff3a3315ecb2345553d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.223939 31678 tablet_bootstrap.cc:654] T 62b3902e073b42b18c3ff62bcdccb2cb P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.224514 31682 tablet_bootstrap.cc:492] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:26.225400 31682 tablet_bootstrap.cc:654] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.225783 31677 tablet_bootstrap.cc:492] T 9f15d8d22c324ff3a3315ecb2345553d P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:26.226379 31678 tablet_bootstrap.cc:492] T 62b3902e073b42b18c3ff62bcdccb2cb P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:26.226455 31678 ts_tablet_manager.cc:1367] T 62b3902e073b42b18c3ff62bcdccb2cb P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:26.226480 30386 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5e1afe21654080b5aa3eb877e8f1d9 (DEFAULT_TABLE table=rebalance_test_table_10 [id=4fa8cc9c35c649efb9c70f46be41dd50]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.226563 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 9f15d8d22c324ff3a3315ecb2345553d (DEFAULT_TABLE table=rebalance_test_table_10 [id=4fa8cc9c35c649efb9c70f46be41dd50]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.226644 31677 tablet_bootstrap.cc:654] T 9f15d8d22c324ff3a3315ecb2345553d P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.226708 30386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5e1afe21654080b5aa3eb877e8f1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.226745 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f15d8d22c324ff3a3315ecb2345553d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.226915 31678 raft_consensus.cc:348] T 62b3902e073b42b18c3ff62bcdccb2cb P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:26.227020 31678 raft_consensus.cc:374] T 62b3902e073b42b18c3ff62bcdccb2cb P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.227051 31678 raft_consensus.cc:729] T 62b3902e073b42b18c3ff62bcdccb2cb P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.227175 31678 consensus_queue.cc:260] T 62b3902e073b42b18c3ff62bcdccb2cb P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:26.227483 31678 ts_tablet_manager.cc:1398] T 62b3902e073b42b18c3ff62bcdccb2cb P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:26.227738 30125 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5e1afe21654080b5aa3eb877e8f1d9 (DEFAULT_TABLE table=rebalance_test_table_10 [id=4fa8cc9c35c649efb9c70f46be41dd50]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.227946 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5e1afe21654080b5aa3eb877e8f1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.229264 31677 tablet_bootstrap.cc:492] T 9f15d8d22c324ff3a3315ecb2345553d P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:26.229223 30387 tablet_service.cc:1467] Processing CreateTablet for tablet 9f15d8d22c324ff3a3315ecb2345553d (DEFAULT_TABLE table=rebalance_test_table_10 [id=4fa8cc9c35c649efb9c70f46be41dd50]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.229347 31677 ts_tablet_manager.cc:1367] T 9f15d8d22c324ff3a3315ecb2345553d P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:26.229431 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f15d8d22c324ff3a3315ecb2345553d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.229764 31678 tablet_bootstrap.cc:492] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:26.229776 31677 raft_consensus.cc:348] T 9f15d8d22c324ff3a3315ecb2345553d P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:26.229877 31677 raft_consensus.cc:374] T 9f15d8d22c324ff3a3315ecb2345553d P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.229907 31677 raft_consensus.cc:729] T 9f15d8d22c324ff3a3315ecb2345553d P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.230015 31677 consensus_queue.cc:260] T 9f15d8d22c324ff3a3315ecb2345553d P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:26.230320 31677 ts_tablet_manager.cc:1398] T 9f15d8d22c324ff3a3315ecb2345553d P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:26.230475 31678 tablet_bootstrap.cc:654] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.231614 31682 tablet_bootstrap.cc:492] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:26.231698 31682 ts_tablet_manager.cc:1367] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:26.232033 31678 tablet_bootstrap.cc:492] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:26.232105 31678 ts_tablet_manager.cc:1367] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:09:26.232113 31682 raft_consensus.cc:348] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:26.232220 31682 raft_consensus.cc:374] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.232297 31682 raft_consensus.cc:729] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.232424 31682 consensus_queue.cc:260] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:26.232699 31682 ts_tablet_manager.cc:1398] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:26.232673 31678 raft_consensus.cc:348] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.232764 31678 raft_consensus.cc:374] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.232795 31678 raft_consensus.cc:729] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.232849 31682 tablet_bootstrap.cc:492] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:26.232864 31676 tablet_bootstrap.cc:492] T 62b3902e073b42b18c3ff62bcdccb2cb P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:26.232939 31676 ts_tablet_manager.cc:1367] T 62b3902e073b42b18c3ff62bcdccb2cb P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20240501 14:09:26.232901 31678 consensus_queue.cc:260] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.233168 31678 ts_tablet_manager.cc:1398] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:26.233304 31678 tablet_bootstrap.cc:492] T 9f15d8d22c324ff3a3315ecb2345553d P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:26.233402 31676 raft_consensus.cc:348] T 62b3902e073b42b18c3ff62bcdccb2cb P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:26.233505 31676 raft_consensus.cc:374] T 62b3902e073b42b18c3ff62bcdccb2cb P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.233541 31676 raft_consensus.cc:729] T 62b3902e073b42b18c3ff62bcdccb2cb P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.233654 31676 consensus_queue.cc:260] T 62b3902e073b42b18c3ff62bcdccb2cb P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:26.233712 31682 tablet_bootstrap.cc:654] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.233911 31676 ts_tablet_manager.cc:1398] T 62b3902e073b42b18c3ff62bcdccb2cb P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:26.234069 31676 tablet_bootstrap.cc:492] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:26.234151 31678 tablet_bootstrap.cc:654] T 9f15d8d22c324ff3a3315ecb2345553d P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.234841 31682 tablet_bootstrap.cc:492] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:26.234920 31682 ts_tablet_manager.cc:1367] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:26.234938 31676 tablet_bootstrap.cc:654] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.235309 31682 raft_consensus.cc:348] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.235394 31682 raft_consensus.cc:374] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.235422 31682 raft_consensus.cc:729] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.235517 31682 consensus_queue.cc:260] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.235800 31682 ts_tablet_manager.cc:1398] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:26.235909 31678 tablet_bootstrap.cc:492] T 9f15d8d22c324ff3a3315ecb2345553d P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:26.235946 31682 tablet_bootstrap.cc:492] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:26.235988 31678 ts_tablet_manager.cc:1367] T 9f15d8d22c324ff3a3315ecb2345553d P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:26.236414 31678 raft_consensus.cc:348] T 9f15d8d22c324ff3a3315ecb2345553d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:26.236477 31678 raft_consensus.cc:374] T 9f15d8d22c324ff3a3315ecb2345553d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.236498 31678 raft_consensus.cc:729] T 9f15d8d22c324ff3a3315ecb2345553d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.236608 31678 consensus_queue.cc:260] T 9f15d8d22c324ff3a3315ecb2345553d P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:26.236858 31682 tablet_bootstrap.cc:654] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.236881 31678 ts_tablet_manager.cc:1398] T 9f15d8d22c324ff3a3315ecb2345553d P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:26.237898 31676 tablet_bootstrap.cc:492] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:26.237947 31682 tablet_bootstrap.cc:492] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:26.237995 31676 ts_tablet_manager.cc:1367] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:09:26.238005 31682 ts_tablet_manager.cc:1367] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:26.238412 31682 raft_consensus.cc:348] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:26.238412 31676 raft_consensus.cc:348] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.238512 31676 raft_consensus.cc:374] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.238512 31682 raft_consensus.cc:374] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.238554 31676 raft_consensus.cc:729] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.238554 31682 raft_consensus.cc:729] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.238668 31676 consensus_queue.cc:260] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.238668 31682 consensus_queue.cc:260] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:26.238895 31682 ts_tablet_manager.cc:1398] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:26.238951 31676 ts_tablet_manager.cc:1398] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:26.342386 31611 raft_consensus.cc:482] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:26.342504 31611 raft_consensus.cc:504] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:26.343005 31611 leader_election.cc:288] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:26.343391 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62b3902e073b42b18c3ff62bcdccb2cb" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:26.343564 30145 raft_consensus.cc:2454] T 62b3902e073b42b18c3ff62bcdccb2cb P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:26.343572 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62b3902e073b42b18c3ff62bcdccb2cb" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:26.343782 30014 raft_consensus.cc:2454] T 62b3902e073b42b18c3ff62bcdccb2cb P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:26.343879 30344 leader_election.cc:302] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:26.344071 31611 raft_consensus.cc:2781] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:26.344130 31611 raft_consensus.cc:482] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:26.344158 31611 raft_consensus.cc:3037] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.345011 31611 raft_consensus.cc:504] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:26.345306 31611 leader_election.cc:288] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:26.345609 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62b3902e073b42b18c3ff62bcdccb2cb" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:26.345633 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62b3902e073b42b18c3ff62bcdccb2cb" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:26.345700 30145 raft_consensus.cc:3037] T 62b3902e073b42b18c3ff62bcdccb2cb P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.345773 30014 raft_consensus.cc:3037] T 62b3902e073b42b18c3ff62bcdccb2cb P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.346450 30145 raft_consensus.cc:2454] T 62b3902e073b42b18c3ff62bcdccb2cb P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:26.346562 30014 raft_consensus.cc:2454] T 62b3902e073b42b18c3ff62bcdccb2cb P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:26.346709 30344 leader_election.cc:302] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:26.346868 31611 raft_consensus.cc:2781] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:26.346961 31611 raft_consensus.cc:686] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:26.347080 31611 consensus_queue.cc:237] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:26.348858 29867 catalog_manager.cc:5484] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:26.395782 31718 raft_consensus.cc:482] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:26.395900 31718 raft_consensus.cc:504] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.396343 31718 leader_election.cc:288] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:26.396636 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5e1afe21654080b5aa3eb877e8f1d9" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:26.396700 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5e1afe21654080b5aa3eb877e8f1d9" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:26.396778 30407 raft_consensus.cc:2454] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:26.396828 30014 raft_consensus.cc:2454] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:26.397053 30079 leader_election.cc:302] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:26.397193 31718 raft_consensus.cc:2781] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:26.397249 31718 raft_consensus.cc:482] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:26.397296 31718 raft_consensus.cc:3037] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.398243 31718 raft_consensus.cc:504] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.398542 31718 leader_election.cc:288] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:26.398828 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5e1afe21654080b5aa3eb877e8f1d9" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:26.398928 30407 raft_consensus.cc:3037] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.398886 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5e1afe21654080b5aa3eb877e8f1d9" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:26.399035 30014 raft_consensus.cc:3037] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.399658 30407 raft_consensus.cc:2454] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:26.399765 30014 raft_consensus.cc:2454] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:26.399993 30079 leader_election.cc:302] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:26.400177 31718 raft_consensus.cc:2781] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:26.400293 31718 raft_consensus.cc:686] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:26.400427 31718 consensus_queue.cc:237] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.402211 29867 catalog_manager.cc:5484] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:26.509616 31611 raft_consensus.cc:482] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:26.509737 31611 raft_consensus.cc:504] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:26.510155 31611 leader_election.cc:288] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:26.510488 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f15d8d22c324ff3a3315ecb2345553d" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:26.510502 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f15d8d22c324ff3a3315ecb2345553d" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:26.510601 30014 raft_consensus.cc:2454] T 9f15d8d22c324ff3a3315ecb2345553d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:26.510617 30276 raft_consensus.cc:2454] T 9f15d8d22c324ff3a3315ecb2345553d P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:26.510844 30343 leader_election.cc:302] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:26.511016 31238 raft_consensus.cc:2781] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:26.511073 31238 raft_consensus.cc:482] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:26.511101 31238 raft_consensus.cc:3037] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.512013 31238 raft_consensus.cc:504] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:26.512327 31238 leader_election.cc:288] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:26.512591 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f15d8d22c324ff3a3315ecb2345553d" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:26.512658 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f15d8d22c324ff3a3315ecb2345553d" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:26.512706 30014 raft_consensus.cc:3037] T 9f15d8d22c324ff3a3315ecb2345553d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.512787 30276 raft_consensus.cc:3037] T 9f15d8d22c324ff3a3315ecb2345553d P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.513442 30014 raft_consensus.cc:2454] T 9f15d8d22c324ff3a3315ecb2345553d P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:26.513495 30276 raft_consensus.cc:2454] T 9f15d8d22c324ff3a3315ecb2345553d P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:26.513724 30343 leader_election.cc:302] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:26.513872 31238 raft_consensus.cc:2781] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:26.513979 31238 raft_consensus.cc:686] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:26.514124 31238 consensus_queue.cc:237] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:26.515813 29867 catalog_manager.cc:5484] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
W20240501 14:09:26.626870 31741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:26.627074 31741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:26.629963 31741 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240501 14:09:26.713749 30014 raft_consensus.cc:1261] T 62b3902e073b42b18c3ff62bcdccb2cb P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.713842 30145 raft_consensus.cc:1261] T 62b3902e073b42b18c3ff62bcdccb2cb P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.714203 31238 consensus_queue.cc:1035] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:26.714331 31611 consensus_queue.cc:1035] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:26.716748 30276 raft_consensus.cc:1261] T 9f15d8d22c324ff3a3315ecb2345553d P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.717128 31238 consensus_queue.cc:1035] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:26.717690 30013 raft_consensus.cc:1261] T 9f15d8d22c324ff3a3315ecb2345553d P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.718454 31238 consensus_queue.cc:1035] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:26.722430 31773 mvcc.cc:205] Tried to move back new op lower bound from 7022889233266585600 to 7022889232442949632. Current Snapshot: MvccSnapshot[applied={T|T < 7022889233266585600}]
I20240501 14:09:26.725853 30014 raft_consensus.cc:1261] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.726184 31718 consensus_queue.cc:1035] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:26.727923 30407 raft_consensus.cc:1261] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.728315 31718 consensus_queue.cc:1035] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.612 ms
time per row: 0.0558059 ms
I20240501 14:09:26.783803 29866 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:26.789077 30257 tablet_service.cc:1467] Processing CreateTablet for tablet f2bc06e76e2a41429d5625ebf99e5e5c (DEFAULT_TABLE table=rebalance_test_table_11 [id=0ceb652d71d942b2a6e0ef7686e7635b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.789358 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2bc06e76e2a41429d5625ebf99e5e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.789994 30386 tablet_service.cc:1467] Processing CreateTablet for tablet 529f938305214c3ba33f61e06d1acd77 (DEFAULT_TABLE table=rebalance_test_table_11 [id=0ceb652d71d942b2a6e0ef7686e7635b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.790060 30125 tablet_service.cc:1467] Processing CreateTablet for tablet f2bc06e76e2a41429d5625ebf99e5e5c (DEFAULT_TABLE table=rebalance_test_table_11 [id=0ceb652d71d942b2a6e0ef7686e7635b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.790267 30386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 529f938305214c3ba33f61e06d1acd77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.790308 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2bc06e76e2a41429d5625ebf99e5e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.791498 30387 tablet_service.cc:1467] Processing CreateTablet for tablet 7e1895c0656d49c09266cdbd231497cb (DEFAULT_TABLE table=rebalance_test_table_11 [id=0ceb652d71d942b2a6e0ef7686e7635b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.791697 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e1895c0656d49c09266cdbd231497cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.792466 30126 tablet_service.cc:1467] Processing CreateTablet for tablet 529f938305214c3ba33f61e06d1acd77 (DEFAULT_TABLE table=rebalance_test_table_11 [id=0ceb652d71d942b2a6e0ef7686e7635b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.792681 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 529f938305214c3ba33f61e06d1acd77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.793815 31803 tablet_bootstrap.cc:492] T 529f938305214c3ba33f61e06d1acd77 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:26.794358 31802 tablet_bootstrap.cc:492] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:26.794790 31803 tablet_bootstrap.cc:654] T 529f938305214c3ba33f61e06d1acd77 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.794865 29993 tablet_service.cc:1467] Processing CreateTablet for tablet f2bc06e76e2a41429d5625ebf99e5e5c (DEFAULT_TABLE table=rebalance_test_table_11 [id=0ceb652d71d942b2a6e0ef7686e7635b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.795181 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2bc06e76e2a41429d5625ebf99e5e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.795367 31802 tablet_bootstrap.cc:654] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.795615 30257 tablet_service.cc:1467] Processing CreateTablet for tablet 7e1895c0656d49c09266cdbd231497cb (DEFAULT_TABLE table=rebalance_test_table_11 [id=0ceb652d71d942b2a6e0ef7686e7635b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.795818 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e1895c0656d49c09266cdbd231497cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.797230 31807 tablet_bootstrap.cc:492] T f2bc06e76e2a41429d5625ebf99e5e5c P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:26.797382 30256 tablet_service.cc:1467] Processing CreateTablet for tablet 529f938305214c3ba33f61e06d1acd77 (DEFAULT_TABLE table=rebalance_test_table_11 [id=0ceb652d71d942b2a6e0ef7686e7635b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.797585 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 529f938305214c3ba33f61e06d1acd77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.797819 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 7e1895c0656d49c09266cdbd231497cb (DEFAULT_TABLE table=rebalance_test_table_11 [id=0ceb652d71d942b2a6e0ef7686e7635b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.798002 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e1895c0656d49c09266cdbd231497cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.798095 31807 tablet_bootstrap.cc:654] T f2bc06e76e2a41429d5625ebf99e5e5c P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.799562 31807 tablet_bootstrap.cc:492] T f2bc06e76e2a41429d5625ebf99e5e5c P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:26.799640 31807 ts_tablet_manager.cc:1367] T f2bc06e76e2a41429d5625ebf99e5e5c P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:26.799842 31804 tablet_bootstrap.cc:492] T f2bc06e76e2a41429d5625ebf99e5e5c P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:26.800088 31807 raft_consensus.cc:348] T f2bc06e76e2a41429d5625ebf99e5e5c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.800218 31807 raft_consensus.cc:374] T f2bc06e76e2a41429d5625ebf99e5e5c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.800318 31807 raft_consensus.cc:729] T f2bc06e76e2a41429d5625ebf99e5e5c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.800463 31807 consensus_queue.cc:260] T f2bc06e76e2a41429d5625ebf99e5e5c P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.800730 31804 tablet_bootstrap.cc:654] T f2bc06e76e2a41429d5625ebf99e5e5c P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.801959 31803 tablet_bootstrap.cc:492] T 529f938305214c3ba33f61e06d1acd77 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:26.802021 31804 tablet_bootstrap.cc:492] T f2bc06e76e2a41429d5625ebf99e5e5c P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:26.802058 31803 ts_tablet_manager.cc:1367] T 529f938305214c3ba33f61e06d1acd77 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:26.802125 31804 ts_tablet_manager.cc:1367] T f2bc06e76e2a41429d5625ebf99e5e5c P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:26.802472 31803 raft_consensus.cc:348] T 529f938305214c3ba33f61e06d1acd77 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:26.802503 31804 raft_consensus.cc:348] T f2bc06e76e2a41429d5625ebf99e5e5c P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.802572 31803 raft_consensus.cc:374] T 529f938305214c3ba33f61e06d1acd77 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.802582 31804 raft_consensus.cc:374] T f2bc06e76e2a41429d5625ebf99e5e5c P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.802608 31803 raft_consensus.cc:729] T 529f938305214c3ba33f61e06d1acd77 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.802614 31804 raft_consensus.cc:729] T f2bc06e76e2a41429d5625ebf99e5e5c P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.802727 31803 consensus_queue.cc:260] T 529f938305214c3ba33f61e06d1acd77 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:26.802748 31804 consensus_queue.cc:260] T f2bc06e76e2a41429d5625ebf99e5e5c P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.802985 31804 ts_tablet_manager.cc:1398] T f2bc06e76e2a41429d5625ebf99e5e5c P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:26.803014 31803 ts_tablet_manager.cc:1398] T 529f938305214c3ba33f61e06d1acd77 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:26.803164 31804 tablet_bootstrap.cc:492] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:26.803164 31803 tablet_bootstrap.cc:492] T 7e1895c0656d49c09266cdbd231497cb P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:26.803942 31804 tablet_bootstrap.cc:654] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.803952 31803 tablet_bootstrap.cc:654] T 7e1895c0656d49c09266cdbd231497cb P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.804508 31807 ts_tablet_manager.cc:1398] T f2bc06e76e2a41429d5625ebf99e5e5c P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:26.805754 31803 tablet_bootstrap.cc:492] T 7e1895c0656d49c09266cdbd231497cb P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:26.805842 31803 ts_tablet_manager.cc:1367] T 7e1895c0656d49c09266cdbd231497cb P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:26.806243 31803 raft_consensus.cc:348] T 7e1895c0656d49c09266cdbd231497cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.806340 31803 raft_consensus.cc:374] T 7e1895c0656d49c09266cdbd231497cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.806370 31803 raft_consensus.cc:729] T 7e1895c0656d49c09266cdbd231497cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.806480 31803 consensus_queue.cc:260] T 7e1895c0656d49c09266cdbd231497cb P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.806741 31803 ts_tablet_manager.cc:1398] T 7e1895c0656d49c09266cdbd231497cb P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:26.806790 31807 tablet_bootstrap.cc:492] T 7e1895c0656d49c09266cdbd231497cb P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:26.807308 31804 tablet_bootstrap.cc:492] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:26.807379 31804 ts_tablet_manager.cc:1367] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:26.807715 31807 tablet_bootstrap.cc:654] T 7e1895c0656d49c09266cdbd231497cb P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.807816 31804 raft_consensus.cc:348] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:26.807910 31804 raft_consensus.cc:374] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.807960 31804 raft_consensus.cc:729] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.808066 31804 consensus_queue.cc:260] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:26.808370 31804 ts_tablet_manager.cc:1398] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:26.808753 31802 tablet_bootstrap.cc:492] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:26.808831 31802 ts_tablet_manager.cc:1367] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 14:09:26.809141 31802 raft_consensus.cc:348] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.809226 31802 raft_consensus.cc:374] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.809249 31802 raft_consensus.cc:729] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.809342 31802 consensus_queue.cc:260] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.809543 31802 ts_tablet_manager.cc:1398] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:26.809654 31802 tablet_bootstrap.cc:492] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:26.810307 31802 tablet_bootstrap.cc:654] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.811374 31802 tablet_bootstrap.cc:492] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:26.811450 31802 ts_tablet_manager.cc:1367] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:26.811636 31807 tablet_bootstrap.cc:492] T 7e1895c0656d49c09266cdbd231497cb P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:26.811694 31807 ts_tablet_manager.cc:1367] T 7e1895c0656d49c09266cdbd231497cb P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:26.811852 31802 raft_consensus.cc:348] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.811955 31802 raft_consensus.cc:374] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.811990 31802 raft_consensus.cc:729] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.812048 31807 raft_consensus.cc:348] T 7e1895c0656d49c09266cdbd231497cb P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.812127 31807 raft_consensus.cc:374] T 7e1895c0656d49c09266cdbd231497cb P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.812111 31802 consensus_queue.cc:260] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.812153 31807 raft_consensus.cc:729] T 7e1895c0656d49c09266cdbd231497cb P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.812376 31802 ts_tablet_manager.cc:1398] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:26.812417 31807 consensus_queue.cc:260] T 7e1895c0656d49c09266cdbd231497cb P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.812528 31802 tablet_bootstrap.cc:492] T 529f938305214c3ba33f61e06d1acd77 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:26.813311 31802 tablet_bootstrap.cc:654] T 529f938305214c3ba33f61e06d1acd77 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.814904 31807 ts_tablet_manager.cc:1398] T 7e1895c0656d49c09266cdbd231497cb P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:26.816766 31802 tablet_bootstrap.cc:492] T 529f938305214c3ba33f61e06d1acd77 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:26.816830 31802 ts_tablet_manager.cc:1367] T 529f938305214c3ba33f61e06d1acd77 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:26.817157 31802 raft_consensus.cc:348] T 529f938305214c3ba33f61e06d1acd77 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:26.817234 31802 raft_consensus.cc:374] T 529f938305214c3ba33f61e06d1acd77 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.817266 31802 raft_consensus.cc:729] T 529f938305214c3ba33f61e06d1acd77 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.817386 31802 consensus_queue.cc:260] T 529f938305214c3ba33f61e06d1acd77 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:26.817624 31802 ts_tablet_manager.cc:1398] T 529f938305214c3ba33f61e06d1acd77 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:26.849090 31718 raft_consensus.cc:482] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:26.849215 31718 raft_consensus.cc:504] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:26.849587 31718 leader_election.cc:288] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:26.849848 31541 raft_consensus.cc:482] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:26.849856 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "529f938305214c3ba33f61e06d1acd77" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:26.849952 31541 raft_consensus.cc:504] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.849953 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "529f938305214c3ba33f61e06d1acd77" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:26.850034 30276 raft_consensus.cc:2454] T 529f938305214c3ba33f61e06d1acd77 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:26.850070 30407 raft_consensus.cc:2454] T 529f938305214c3ba33f61e06d1acd77 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:26.850317 30081 leader_election.cc:302] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:26.850342 31541 leader_election.cc:288] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:26.850685 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bc06e76e2a41429d5625ebf99e5e5c" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:26.850807 30145 raft_consensus.cc:2454] T f2bc06e76e2a41429d5625ebf99e5e5c P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:26.850968 31718 raft_consensus.cc:2781] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:26.851024 31718 raft_consensus.cc:482] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:26.850973 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bc06e76e2a41429d5625ebf99e5e5c" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:26.851064 31718 raft_consensus.cc:3037] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.851126 30013 raft_consensus.cc:2454] T f2bc06e76e2a41429d5625ebf99e5e5c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:26.851328 30213 leader_election.cc:302] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:26.851600 31541 raft_consensus.cc:2781] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:26.851655 31541 raft_consensus.cc:482] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:26.851694 31541 raft_consensus.cc:3037] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.852061 31718 raft_consensus.cc:504] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:26.852413 31718 leader_election.cc:288] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:26.852691 31541 raft_consensus.cc:504] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.852696 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "529f938305214c3ba33f61e06d1acd77" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:26.852777 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "529f938305214c3ba33f61e06d1acd77" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:26.852845 30276 raft_consensus.cc:3037] T 529f938305214c3ba33f61e06d1acd77 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.852869 30407 raft_consensus.cc:3037] T 529f938305214c3ba33f61e06d1acd77 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.853029 31541 leader_election.cc:288] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:26.853706 30276 raft_consensus.cc:2454] T 529f938305214c3ba33f61e06d1acd77 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:26.853744 30407 raft_consensus.cc:2454] T 529f938305214c3ba33f61e06d1acd77 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:26.854084 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bc06e76e2a41429d5625ebf99e5e5c" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:26.854192 30013 raft_consensus.cc:3037] T f2bc06e76e2a41429d5625ebf99e5e5c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.854182 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2bc06e76e2a41429d5625ebf99e5e5c" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:26.854182 30079 leader_election.cc:302] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:26.854312 30145 raft_consensus.cc:3037] T f2bc06e76e2a41429d5625ebf99e5e5c P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.854391 31718 raft_consensus.cc:2781] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:26.854475 31718 raft_consensus.cc:686] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:26.854590 31718 consensus_queue.cc:237] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:26.855190 30013 raft_consensus.cc:2454] T f2bc06e76e2a41429d5625ebf99e5e5c P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:26.855191 30145 raft_consensus.cc:2454] T f2bc06e76e2a41429d5625ebf99e5e5c P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:26.855430 30213 leader_election.cc:302] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:26.855703 31815 raft_consensus.cc:2781] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:26.855770 31815 raft_consensus.cc:686] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:26.855893 31815 consensus_queue.cc:237] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:26.856813 29867 catalog_manager.cc:5484] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:26.857700 29866 catalog_manager.cc:5484] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:27.163849 31541 raft_consensus.cc:482] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:27.163964 31541 raft_consensus.cc:504] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:27.164520 31541 leader_election.cc:288] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:27.164775 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e1895c0656d49c09266cdbd231497cb" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:27.164831 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e1895c0656d49c09266cdbd231497cb" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:27.164942 30407 raft_consensus.cc:2454] T 7e1895c0656d49c09266cdbd231497cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:27.164971 30013 raft_consensus.cc:2454] T 7e1895c0656d49c09266cdbd231497cb P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:27.165298 30210 leader_election.cc:302] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:27.165480 31541 raft_consensus.cc:2781] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:27.165542 31541 raft_consensus.cc:482] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:27.165591 31541 raft_consensus.cc:3037] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.166368 31541 raft_consensus.cc:504] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:27.166658 31541 leader_election.cc:288] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:27.166914 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e1895c0656d49c09266cdbd231497cb" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:27.166923 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e1895c0656d49c09266cdbd231497cb" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:27.167003 30013 raft_consensus.cc:3037] T 7e1895c0656d49c09266cdbd231497cb P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.167011 30407 raft_consensus.cc:3037] T 7e1895c0656d49c09266cdbd231497cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.167703 30013 raft_consensus.cc:2454] T 7e1895c0656d49c09266cdbd231497cb P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:27.167716 30407 raft_consensus.cc:2454] T 7e1895c0656d49c09266cdbd231497cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:27.167932 30212 leader_election.cc:302] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:27.168081 31541 raft_consensus.cc:2781] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:27.168154 31541 raft_consensus.cc:686] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:27.168321 31541 consensus_queue.cc:237] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:27.170118 29866 catalog_manager.cc:5484] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:27.241401 31718 consensus_queue.cc:1035] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:27.244585 31718 consensus_queue.cc:1035] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:27.256299 31541 consensus_queue.cc:1035] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:27.259167 31823 consensus_queue.cc:1035] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:27.353977 31820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:27.354246 31820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:27.357270 31820 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240501 14:09:27.438565 30013 raft_consensus.cc:1261] T 7e1895c0656d49c09266cdbd231497cb P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:27.438881 30407 raft_consensus.cc:1261] T 7e1895c0656d49c09266cdbd231497cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:27.439271 31541 consensus_queue.cc:1035] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:27.439399 31823 consensus_queue.cc:1035] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 108.559 ms
time per row: 0.0542795 ms
I20240501 14:09:27.504122 29867 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:27.509158 30125 tablet_service.cc:1467] Processing CreateTablet for tablet aa03df5f44864c9eb0a642f0481def39 (DEFAULT_TABLE table=rebalance_test_table_12 [id=b37dbfc47a574f218dd59bef604c1bc1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.509496 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa03df5f44864c9eb0a642f0481def39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.509413 30256 tablet_service.cc:1467] Processing CreateTablet for tablet aa03df5f44864c9eb0a642f0481def39 (DEFAULT_TABLE table=rebalance_test_table_12 [id=b37dbfc47a574f218dd59bef604c1bc1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.509698 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa03df5f44864c9eb0a642f0481def39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.509656 30126 tablet_service.cc:1467] Processing CreateTablet for tablet 354658866eda4a2da6a2871fe0a2769d (DEFAULT_TABLE table=rebalance_test_table_12 [id=b37dbfc47a574f218dd59bef604c1bc1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.509881 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354658866eda4a2da6a2871fe0a2769d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.510313 30257 tablet_service.cc:1467] Processing CreateTablet for tablet 92cac7a9b26444e3916d4d7e5dd0032b (DEFAULT_TABLE table=rebalance_test_table_12 [id=b37dbfc47a574f218dd59bef604c1bc1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.510538 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92cac7a9b26444e3916d4d7e5dd0032b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.512073 29993 tablet_service.cc:1467] Processing CreateTablet for tablet aa03df5f44864c9eb0a642f0481def39 (DEFAULT_TABLE table=rebalance_test_table_12 [id=b37dbfc47a574f218dd59bef604c1bc1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.512296 30387 tablet_service.cc:1467] Processing CreateTablet for tablet 354658866eda4a2da6a2871fe0a2769d (DEFAULT_TABLE table=rebalance_test_table_12 [id=b37dbfc47a574f218dd59bef604c1bc1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.512368 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa03df5f44864c9eb0a642f0481def39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.512589 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354658866eda4a2da6a2871fe0a2769d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.512933 30386 tablet_service.cc:1467] Processing CreateTablet for tablet 92cac7a9b26444e3916d4d7e5dd0032b (DEFAULT_TABLE table=rebalance_test_table_12 [id=b37dbfc47a574f218dd59bef604c1bc1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.513172 30386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92cac7a9b26444e3916d4d7e5dd0032b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.513991 31877 tablet_bootstrap.cc:492] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:27.514802 29994 tablet_service.cc:1467] Processing CreateTablet for tablet 354658866eda4a2da6a2871fe0a2769d (DEFAULT_TABLE table=rebalance_test_table_12 [id=b37dbfc47a574f218dd59bef604c1bc1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.515049 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354658866eda4a2da6a2871fe0a2769d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.515277 31877 tablet_bootstrap.cc:654] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.515981 31879 tablet_bootstrap.cc:492] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:27.516422 31878 tablet_bootstrap.cc:492] T aa03df5f44864c9eb0a642f0481def39 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:27.516840 31879 tablet_bootstrap.cc:654] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.517308 31878 tablet_bootstrap.cc:654] T aa03df5f44864c9eb0a642f0481def39 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.517586 29994 tablet_service.cc:1467] Processing CreateTablet for tablet 92cac7a9b26444e3916d4d7e5dd0032b (DEFAULT_TABLE table=rebalance_test_table_12 [id=b37dbfc47a574f218dd59bef604c1bc1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.517762 31879 tablet_bootstrap.cc:492] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:27.517799 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92cac7a9b26444e3916d4d7e5dd0032b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.517832 31879 ts_tablet_manager.cc:1367] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:27.518216 31879 raft_consensus.cc:348] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:27.518309 31879 raft_consensus.cc:374] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.518330 31879 raft_consensus.cc:729] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.518393 31878 tablet_bootstrap.cc:492] T aa03df5f44864c9eb0a642f0481def39 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:27.518430 31879 consensus_queue.cc:260] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:27.518468 31878 ts_tablet_manager.cc:1367] T aa03df5f44864c9eb0a642f0481def39 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:27.518675 31879 ts_tablet_manager.cc:1398] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:27.518790 31879 tablet_bootstrap.cc:492] T 92cac7a9b26444e3916d4d7e5dd0032b P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:27.518893 31878 raft_consensus.cc:348] T aa03df5f44864c9eb0a642f0481def39 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:27.518988 31878 raft_consensus.cc:374] T aa03df5f44864c9eb0a642f0481def39 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.519017 31878 raft_consensus.cc:729] T aa03df5f44864c9eb0a642f0481def39 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.519155 31878 consensus_queue.cc:260] T aa03df5f44864c9eb0a642f0481def39 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:27.519382 31879 tablet_bootstrap.cc:654] T 92cac7a9b26444e3916d4d7e5dd0032b P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.519687 31878 ts_tablet_manager.cc:1398] T aa03df5f44864c9eb0a642f0481def39 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:27.519856 31878 tablet_bootstrap.cc:492] T 354658866eda4a2da6a2871fe0a2769d P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:27.519865 31876 tablet_bootstrap.cc:492] T aa03df5f44864c9eb0a642f0481def39 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:27.521274 31879 tablet_bootstrap.cc:492] T 92cac7a9b26444e3916d4d7e5dd0032b P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:27.521328 31876 tablet_bootstrap.cc:654] T aa03df5f44864c9eb0a642f0481def39 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.521350 31879 ts_tablet_manager.cc:1367] T 92cac7a9b26444e3916d4d7e5dd0032b P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:27.521718 31879 raft_consensus.cc:348] T 92cac7a9b26444e3916d4d7e5dd0032b P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:27.521804 31879 raft_consensus.cc:374] T 92cac7a9b26444e3916d4d7e5dd0032b P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.521831 31879 raft_consensus.cc:729] T 92cac7a9b26444e3916d4d7e5dd0032b P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.521920 31879 consensus_queue.cc:260] T 92cac7a9b26444e3916d4d7e5dd0032b P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:27.522166 31879 ts_tablet_manager.cc:1398] T 92cac7a9b26444e3916d4d7e5dd0032b P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:27.522399 31877 tablet_bootstrap.cc:492] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:27.522493 31877 ts_tablet_manager.cc:1367] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:27.522927 31877 raft_consensus.cc:348] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:27.523036 31877 raft_consensus.cc:374] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.523077 31877 raft_consensus.cc:729] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.523190 31877 consensus_queue.cc:260] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:27.523324 31878 tablet_bootstrap.cc:654] T 354658866eda4a2da6a2871fe0a2769d P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.523519 31877 ts_tablet_manager.cc:1398] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:27.523679 31877 tablet_bootstrap.cc:492] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:27.524356 31878 tablet_bootstrap.cc:492] T 354658866eda4a2da6a2871fe0a2769d P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:27.524379 31876 tablet_bootstrap.cc:492] T aa03df5f44864c9eb0a642f0481def39 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:27.524426 31878 ts_tablet_manager.cc:1367] T 354658866eda4a2da6a2871fe0a2769d P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:27.524456 31876 ts_tablet_manager.cc:1367] T aa03df5f44864c9eb0a642f0481def39 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:27.524868 31878 raft_consensus.cc:348] T 354658866eda4a2da6a2871fe0a2769d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:27.524966 31878 raft_consensus.cc:374] T 354658866eda4a2da6a2871fe0a2769d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.525005 31878 raft_consensus.cc:729] T 354658866eda4a2da6a2871fe0a2769d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.525028 31877 tablet_bootstrap.cc:654] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.525029 31876 raft_consensus.cc:348] T aa03df5f44864c9eb0a642f0481def39 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:27.525120 31876 raft_consensus.cc:374] T aa03df5f44864c9eb0a642f0481def39 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.525156 31876 raft_consensus.cc:729] T aa03df5f44864c9eb0a642f0481def39 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.525118 31878 consensus_queue.cc:260] T 354658866eda4a2da6a2871fe0a2769d P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:27.525321 31876 consensus_queue.cc:260] T aa03df5f44864c9eb0a642f0481def39 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:27.525970 31878 ts_tablet_manager.cc:1398] T 354658866eda4a2da6a2871fe0a2769d P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:27.526134 31878 tablet_bootstrap.cc:492] T 92cac7a9b26444e3916d4d7e5dd0032b P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:27.526835 31877 tablet_bootstrap.cc:492] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:27.526913 31877 ts_tablet_manager.cc:1367] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:27.526976 31878 tablet_bootstrap.cc:654] T 92cac7a9b26444e3916d4d7e5dd0032b P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.527316 31877 raft_consensus.cc:348] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:27.527400 31877 raft_consensus.cc:374] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.527427 31877 raft_consensus.cc:729] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.527524 31877 consensus_queue.cc:260] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:27.527674 31876 ts_tablet_manager.cc:1398] T aa03df5f44864c9eb0a642f0481def39 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:27.527776 31877 ts_tablet_manager.cc:1398] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:27.527858 31876 tablet_bootstrap.cc:492] T 354658866eda4a2da6a2871fe0a2769d P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:27.530040 31876 tablet_bootstrap.cc:654] T 354658866eda4a2da6a2871fe0a2769d P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.530972 31878 tablet_bootstrap.cc:492] T 92cac7a9b26444e3916d4d7e5dd0032b P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:27.531049 31878 ts_tablet_manager.cc:1367] T 92cac7a9b26444e3916d4d7e5dd0032b P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240501 14:09:27.531106 31876 tablet_bootstrap.cc:492] T 354658866eda4a2da6a2871fe0a2769d P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:27.531189 31876 ts_tablet_manager.cc:1367] T 354658866eda4a2da6a2871fe0a2769d P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:27.531441 31878 raft_consensus.cc:348] T 92cac7a9b26444e3916d4d7e5dd0032b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:27.531558 31878 raft_consensus.cc:374] T 92cac7a9b26444e3916d4d7e5dd0032b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.531530 31876 raft_consensus.cc:348] T 354658866eda4a2da6a2871fe0a2769d P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:27.531600 31878 raft_consensus.cc:729] T 92cac7a9b26444e3916d4d7e5dd0032b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.531591 31874 raft_consensus.cc:482] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:27.531630 31876 raft_consensus.cc:374] T 354658866eda4a2da6a2871fe0a2769d P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.531661 31876 raft_consensus.cc:729] T 354658866eda4a2da6a2871fe0a2769d P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.531668 31874 raft_consensus.cc:504] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:27.531745 31878 consensus_queue.cc:260] T 92cac7a9b26444e3916d4d7e5dd0032b P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:27.531771 31876 consensus_queue.cc:260] T 354658866eda4a2da6a2871fe0a2769d P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:27.532019 31878 ts_tablet_manager.cc:1398] T 92cac7a9b26444e3916d4d7e5dd0032b P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:27.532022 31874 leader_election.cc:288] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:27.532044 31876 ts_tablet_manager.cc:1398] T 354658866eda4a2da6a2871fe0a2769d P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:27.532447 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354658866eda4a2da6a2871fe0a2769d" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:27.532542 30013 raft_consensus.cc:2454] T 354658866eda4a2da6a2871fe0a2769d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:27.532539 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354658866eda4a2da6a2871fe0a2769d" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:27.532645 30145 raft_consensus.cc:2454] T 354658866eda4a2da6a2871fe0a2769d P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:27.532793 30343 leader_election.cc:302] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:27.533412 31874 raft_consensus.cc:2781] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:27.533463 31874 raft_consensus.cc:482] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:27.533493 31874 raft_consensus.cc:3037] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.534449 31874 raft_consensus.cc:504] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:27.534794 31874 leader_election.cc:288] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:27.535033 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354658866eda4a2da6a2871fe0a2769d" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:27.535107 30013 raft_consensus.cc:3037] T 354658866eda4a2da6a2871fe0a2769d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.535398 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354658866eda4a2da6a2871fe0a2769d" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:27.535501 30145 raft_consensus.cc:3037] T 354658866eda4a2da6a2871fe0a2769d P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.536024 30013 raft_consensus.cc:2454] T 354658866eda4a2da6a2871fe0a2769d P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:27.536311 30343 leader_election.cc:302] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:27.536460 30145 raft_consensus.cc:2454] T 354658866eda4a2da6a2871fe0a2769d P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:27.536638 31874 raft_consensus.cc:2781] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:27.536723 31874 raft_consensus.cc:686] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:27.536855 31874 consensus_queue.cc:237] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:27.538549 29867 catalog_manager.cc:5484] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:27.583425 31815 raft_consensus.cc:482] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:27.583539 31815 raft_consensus.cc:504] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:27.583971 31815 leader_election.cc:288] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:27.584312 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92cac7a9b26444e3916d4d7e5dd0032b" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:27.584434 30013 raft_consensus.cc:2454] T 92cac7a9b26444e3916d4d7e5dd0032b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:27.584674 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92cac7a9b26444e3916d4d7e5dd0032b" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:27.584723 30212 leader_election.cc:302] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:27.584805 30407 raft_consensus.cc:2454] T 92cac7a9b26444e3916d4d7e5dd0032b P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:27.584908 31815 raft_consensus.cc:2781] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:27.584980 31815 raft_consensus.cc:482] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:27.585018 31815 raft_consensus.cc:3037] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.585837 31815 raft_consensus.cc:504] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:27.586207 31815 leader_election.cc:288] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:27.586475 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92cac7a9b26444e3916d4d7e5dd0032b" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:27.586560 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92cac7a9b26444e3916d4d7e5dd0032b" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:27.586606 30013 raft_consensus.cc:3037] T 92cac7a9b26444e3916d4d7e5dd0032b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.586681 30407 raft_consensus.cc:3037] T 92cac7a9b26444e3916d4d7e5dd0032b P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.587395 30013 raft_consensus.cc:2454] T 92cac7a9b26444e3916d4d7e5dd0032b P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:27.587615 30407 raft_consensus.cc:2454] T 92cac7a9b26444e3916d4d7e5dd0032b P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:27.587654 30212 leader_election.cc:302] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:27.587812 31815 raft_consensus.cc:2781] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:27.587891 31815 raft_consensus.cc:686] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:27.588042 31815 consensus_queue.cc:237] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:27.590231 29867 catalog_manager.cc:5484] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:27.712838 31815 raft_consensus.cc:482] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:27.712996 31815 raft_consensus.cc:504] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:27.713377 31815 leader_election.cc:288] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:27.713769 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa03df5f44864c9eb0a642f0481def39" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:27.713766 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa03df5f44864c9eb0a642f0481def39" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:27.713912 30013 raft_consensus.cc:2454] T aa03df5f44864c9eb0a642f0481def39 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:27.713927 30145 raft_consensus.cc:2454] T aa03df5f44864c9eb0a642f0481def39 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:27.714221 30212 leader_election.cc:302] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:27.714403 31815 raft_consensus.cc:2781] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:27.714457 31815 raft_consensus.cc:482] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:27.714509 31815 raft_consensus.cc:3037] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.717331 31815 raft_consensus.cc:504] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:27.717710 31815 leader_election.cc:288] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:27.718081 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa03df5f44864c9eb0a642f0481def39" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:27.718070 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa03df5f44864c9eb0a642f0481def39" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:27.718201 30013 raft_consensus.cc:3037] T aa03df5f44864c9eb0a642f0481def39 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.718216 30145 raft_consensus.cc:3037] T aa03df5f44864c9eb0a642f0481def39 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.719156 30013 raft_consensus.cc:2454] T aa03df5f44864c9eb0a642f0481def39 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:27.719156 30145 raft_consensus.cc:2454] T aa03df5f44864c9eb0a642f0481def39 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:27.719446 30213 leader_election.cc:302] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:27.719588 31541 raft_consensus.cc:2781] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:27.719678 31541 raft_consensus.cc:686] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:27.719826 31541 consensus_queue.cc:237] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:27.721597 29867 catalog_manager.cc:5484] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
W20240501 14:09:27.866622 31892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:27.866874 31892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:27.870117 31892 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240501 14:09:27.941062 31784 consensus_queue.cc:1035] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:27.944962 31874 consensus_queue.cc:1035] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:27.966854 30013 raft_consensus.cc:1261] T aa03df5f44864c9eb0a642f0481def39 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:27.967254 30146 raft_consensus.cc:1261] T aa03df5f44864c9eb0a642f0481def39 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:27.968642 31925 consensus_queue.cc:1035] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:27.968793 30407 raft_consensus.cc:1261] T 92cac7a9b26444e3916d4d7e5dd0032b P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:27.968782 31541 consensus_queue.cc:1035] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:27.969543 31925 consensus_queue.cc:1035] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:27.970395 30013 raft_consensus.cc:1261] T 92cac7a9b26444e3916d4d7e5dd0032b P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:27.970760 31541 consensus_queue.cc:1035] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 126.218 ms
time per row: 0.063109 ms
I20240501 14:09:28.039388 29866 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:28.045032 30386 tablet_service.cc:1467] Processing CreateTablet for tablet c362bd01a59b4bcf85eeb59713e58b64 (DEFAULT_TABLE table=rebalance_test_table_13 [id=422856ed458a4c52950e8e3e49019a55]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.045373 30386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c362bd01a59b4bcf85eeb59713e58b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.045686 30387 tablet_service.cc:1467] Processing CreateTablet for tablet 1698ca3ec26741bb9b3131fe67c9682b (DEFAULT_TABLE table=rebalance_test_table_13 [id=422856ed458a4c52950e8e3e49019a55]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.045938 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1698ca3ec26741bb9b3131fe67c9682b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.045994 30257 tablet_service.cc:1467] Processing CreateTablet for tablet c362bd01a59b4bcf85eeb59713e58b64 (DEFAULT_TABLE table=rebalance_test_table_13 [id=422856ed458a4c52950e8e3e49019a55]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.046089 30256 tablet_service.cc:1467] Processing CreateTablet for tablet 1698ca3ec26741bb9b3131fe67c9682b (DEFAULT_TABLE table=rebalance_test_table_13 [id=422856ed458a4c52950e8e3e49019a55]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.046257 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c362bd01a59b4bcf85eeb59713e58b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.046591 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1698ca3ec26741bb9b3131fe67c9682b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.047994 31942 tablet_bootstrap.cc:492] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:28.048522 31943 tablet_bootstrap.cc:492] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:28.049028 31942 tablet_bootstrap.cc:654] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.049185 30125 tablet_service.cc:1467] Processing CreateTablet for tablet c362bd01a59b4bcf85eeb59713e58b64 (DEFAULT_TABLE table=rebalance_test_table_13 [id=422856ed458a4c52950e8e3e49019a55]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.049242 30126 tablet_service.cc:1467] Processing CreateTablet for tablet 1698ca3ec26741bb9b3131fe67c9682b (DEFAULT_TABLE table=rebalance_test_table_13 [id=422856ed458a4c52950e8e3e49019a55]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.049242 30124 tablet_service.cc:1467] Processing CreateTablet for tablet 18918995fb03421b9dfb4d05369c1ce8 (DEFAULT_TABLE table=rebalance_test_table_13 [id=422856ed458a4c52950e8e3e49019a55]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.049468 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c362bd01a59b4bcf85eeb59713e58b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.049744 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18918995fb03421b9dfb4d05369c1ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.050498 29994 tablet_service.cc:1467] Processing CreateTablet for tablet 18918995fb03421b9dfb4d05369c1ce8 (DEFAULT_TABLE table=rebalance_test_table_13 [id=422856ed458a4c52950e8e3e49019a55]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.050784 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18918995fb03421b9dfb4d05369c1ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.051430 31945 tablet_bootstrap.cc:492] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:28.051913 31942 tablet_bootstrap.cc:492] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:28.051983 31942 ts_tablet_manager.cc:1367] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:28.052171 31943 tablet_bootstrap.cc:654] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.052386 31942 raft_consensus.cc:348] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:28.052469 31942 raft_consensus.cc:374] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.052488 31942 raft_consensus.cc:729] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.052596 31942 consensus_queue.cc:260] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:28.052846 31942 ts_tablet_manager.cc:1398] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:28.053033 31942 tablet_bootstrap.cc:492] T 1698ca3ec26741bb9b3131fe67c9682b P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:28.053319 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1698ca3ec26741bb9b3131fe67c9682b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.054000 31942 tablet_bootstrap.cc:654] T 1698ca3ec26741bb9b3131fe67c9682b P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.055289 31945 tablet_bootstrap.cc:654] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.055640 30256 tablet_service.cc:1467] Processing CreateTablet for tablet 18918995fb03421b9dfb4d05369c1ce8 (DEFAULT_TABLE table=rebalance_test_table_13 [id=422856ed458a4c52950e8e3e49019a55]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.055871 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18918995fb03421b9dfb4d05369c1ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.056279 31942 tablet_bootstrap.cc:492] T 1698ca3ec26741bb9b3131fe67c9682b P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:28.056427 31942 ts_tablet_manager.cc:1367] T 1698ca3ec26741bb9b3131fe67c9682b P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:28.056545 31945 tablet_bootstrap.cc:492] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:28.056630 31945 ts_tablet_manager.cc:1367] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:28.057078 31945 raft_consensus.cc:348] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:28.057181 31945 raft_consensus.cc:374] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.057214 31945 raft_consensus.cc:729] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.057348 31945 consensus_queue.cc:260] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:28.057648 31945 ts_tablet_manager.cc:1398] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:28.057849 31945 tablet_bootstrap.cc:492] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:28.058382 31947 tablet_bootstrap.cc:492] T 18918995fb03421b9dfb4d05369c1ce8 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:28.058563 31942 raft_consensus.cc:348] T 1698ca3ec26741bb9b3131fe67c9682b P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:28.058672 31942 raft_consensus.cc:374] T 1698ca3ec26741bb9b3131fe67c9682b P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.058713 31942 raft_consensus.cc:729] T 1698ca3ec26741bb9b3131fe67c9682b P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.058738 31945 tablet_bootstrap.cc:654] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.058842 31942 consensus_queue.cc:260] T 1698ca3ec26741bb9b3131fe67c9682b P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:28.059167 31947 tablet_bootstrap.cc:654] T 18918995fb03421b9dfb4d05369c1ce8 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.059744 31942 ts_tablet_manager.cc:1398] T 1698ca3ec26741bb9b3131fe67c9682b P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:28.060086 31947 tablet_bootstrap.cc:492] T 18918995fb03421b9dfb4d05369c1ce8 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:28.060145 31947 ts_tablet_manager.cc:1367] T 18918995fb03421b9dfb4d05369c1ce8 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:28.060758 31945 tablet_bootstrap.cc:492] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:28.060726 31947 raft_consensus.cc:348] T 18918995fb03421b9dfb4d05369c1ce8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:28.060834 31945 ts_tablet_manager.cc:1367] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:28.060827 31947 raft_consensus.cc:374] T 18918995fb03421b9dfb4d05369c1ce8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.060870 31947 raft_consensus.cc:729] T 18918995fb03421b9dfb4d05369c1ce8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.061034 31947 consensus_queue.cc:260] T 18918995fb03421b9dfb4d05369c1ce8 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:28.061254 31945 raft_consensus.cc:348] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:28.061326 31947 ts_tablet_manager.cc:1398] T 18918995fb03421b9dfb4d05369c1ce8 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:28.061342 31945 raft_consensus.cc:374] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.061365 31945 raft_consensus.cc:729] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.061810 31943 tablet_bootstrap.cc:492] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:28.061836 31945 consensus_queue.cc:260] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:28.061902 31943 ts_tablet_manager.cc:1367] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:09:28.062168 31945 ts_tablet_manager.cc:1398] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:28.062328 31945 tablet_bootstrap.cc:492] T 1698ca3ec26741bb9b3131fe67c9682b P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:28.062361 31943 raft_consensus.cc:348] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:28.062454 31943 raft_consensus.cc:374] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.062484 31943 raft_consensus.cc:729] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.062592 31943 consensus_queue.cc:260] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:28.062865 31943 ts_tablet_manager.cc:1398] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:28.063026 31943 tablet_bootstrap.cc:492] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:28.063148 31945 tablet_bootstrap.cc:654] T 1698ca3ec26741bb9b3131fe67c9682b P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.063900 31943 tablet_bootstrap.cc:654] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.064270 31945 tablet_bootstrap.cc:492] T 1698ca3ec26741bb9b3131fe67c9682b P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:28.064342 31945 ts_tablet_manager.cc:1367] T 1698ca3ec26741bb9b3131fe67c9682b P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:28.064736 31945 raft_consensus.cc:348] T 1698ca3ec26741bb9b3131fe67c9682b P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:28.064822 31945 raft_consensus.cc:374] T 1698ca3ec26741bb9b3131fe67c9682b P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.064852 31945 raft_consensus.cc:729] T 1698ca3ec26741bb9b3131fe67c9682b P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.064954 31945 consensus_queue.cc:260] T 1698ca3ec26741bb9b3131fe67c9682b P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:28.065318 31945 ts_tablet_manager.cc:1398] T 1698ca3ec26741bb9b3131fe67c9682b P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:28.067377 31943 tablet_bootstrap.cc:492] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:28.067459 31943 ts_tablet_manager.cc:1367] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:09:28.067864 31943 raft_consensus.cc:348] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:28.067977 31943 raft_consensus.cc:374] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.068008 31943 raft_consensus.cc:729] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.068120 31943 consensus_queue.cc:260] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:28.068406 31943 ts_tablet_manager.cc:1398] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:28.068531 31943 tablet_bootstrap.cc:492] T 18918995fb03421b9dfb4d05369c1ce8 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:28.069254 31943 tablet_bootstrap.cc:654] T 18918995fb03421b9dfb4d05369c1ce8 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.070215 31943 tablet_bootstrap.cc:492] T 18918995fb03421b9dfb4d05369c1ce8 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:28.070283 31943 ts_tablet_manager.cc:1367] T 18918995fb03421b9dfb4d05369c1ce8 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:28.070601 31943 raft_consensus.cc:348] T 18918995fb03421b9dfb4d05369c1ce8 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:28.070686 31943 raft_consensus.cc:374] T 18918995fb03421b9dfb4d05369c1ce8 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.070719 31943 raft_consensus.cc:729] T 18918995fb03421b9dfb4d05369c1ce8 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.070835 31943 consensus_queue.cc:260] T 18918995fb03421b9dfb4d05369c1ce8 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:28.071066 31943 ts_tablet_manager.cc:1398] T 18918995fb03421b9dfb4d05369c1ce8 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:28.133244 31061 raft_consensus.cc:482] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:28.133361 31061 raft_consensus.cc:504] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:28.133735 31061 leader_election.cc:288] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:28.134095 30015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18918995fb03421b9dfb4d05369c1ce8" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:28.134168 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18918995fb03421b9dfb4d05369c1ce8" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:28.134223 30015 raft_consensus.cc:2454] T 18918995fb03421b9dfb4d05369c1ce8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:28.134337 30276 raft_consensus.cc:2454] T 18918995fb03421b9dfb4d05369c1ce8 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:28.134516 30081 leader_election.cc:302] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:28.134711 31061 raft_consensus.cc:2781] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:28.134773 31061 raft_consensus.cc:482] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:28.134819 31061 raft_consensus.cc:3037] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.135633 31061 raft_consensus.cc:504] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:28.135942 31061 leader_election.cc:288] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:28.136265 30015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18918995fb03421b9dfb4d05369c1ce8" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:28.136349 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18918995fb03421b9dfb4d05369c1ce8" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:28.136390 30015 raft_consensus.cc:3037] T 18918995fb03421b9dfb4d05369c1ce8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.136463 30276 raft_consensus.cc:3037] T 18918995fb03421b9dfb4d05369c1ce8 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.137393 30276 raft_consensus.cc:2454] T 18918995fb03421b9dfb4d05369c1ce8 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:28.137393 30015 raft_consensus.cc:2454] T 18918995fb03421b9dfb4d05369c1ce8 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:28.137653 30081 leader_election.cc:302] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:28.137830 31061 raft_consensus.cc:2781] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:28.137913 31061 raft_consensus.cc:686] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:28.138053 31061 consensus_queue.cc:237] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:28.139719 29868 catalog_manager.cc:5484] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:28.158519 31541 raft_consensus.cc:482] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:28.158623 31541 raft_consensus.cc:504] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:28.158962 31541 leader_election.cc:288] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:28.159271 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1698ca3ec26741bb9b3131fe67c9682b" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:28.159271 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1698ca3ec26741bb9b3131fe67c9682b" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:28.159415 30146 raft_consensus.cc:2454] T 1698ca3ec26741bb9b3131fe67c9682b P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:28.159410 30407 raft_consensus.cc:2454] T 1698ca3ec26741bb9b3131fe67c9682b P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:28.159714 30210 leader_election.cc:302] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:28.159884 31541 raft_consensus.cc:2781] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:28.159932 31541 raft_consensus.cc:482] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:28.159996 31541 raft_consensus.cc:3037] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.161115 31541 raft_consensus.cc:504] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:28.161429 31541 leader_election.cc:288] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:28.161722 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1698ca3ec26741bb9b3131fe67c9682b" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:28.161722 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1698ca3ec26741bb9b3131fe67c9682b" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:28.161844 30407 raft_consensus.cc:3037] T 1698ca3ec26741bb9b3131fe67c9682b P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.161839 30146 raft_consensus.cc:3037] T 1698ca3ec26741bb9b3131fe67c9682b P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.162796 30407 raft_consensus.cc:2454] T 1698ca3ec26741bb9b3131fe67c9682b P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:28.162796 30146 raft_consensus.cc:2454] T 1698ca3ec26741bb9b3131fe67c9682b P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:28.163062 30213 leader_election.cc:302] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:28.163237 31541 raft_consensus.cc:2781] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:28.163342 31541 raft_consensus.cc:686] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:28.163467 31541 consensus_queue.cc:237] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:28.165207 29868 catalog_manager.cc:5484] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:28.331573 31061 raft_consensus.cc:482] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:28.331687 31061 raft_consensus.cc:504] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:28.332074 31061 leader_election.cc:288] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:28.332402 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c362bd01a59b4bcf85eeb59713e58b64" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:28.332446 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c362bd01a59b4bcf85eeb59713e58b64" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:28.332551 30276 raft_consensus.cc:2454] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:28.332578 30407 raft_consensus.cc:2454] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:28.332824 30079 leader_election.cc:302] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:28.332998 31061 raft_consensus.cc:2781] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:28.333053 31061 raft_consensus.cc:482] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:28.333081 31061 raft_consensus.cc:3037] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.334025 31061 raft_consensus.cc:504] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:28.334367 31061 leader_election.cc:288] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:28.334563 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c362bd01a59b4bcf85eeb59713e58b64" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:28.334681 30407 raft_consensus.cc:3037] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.334671 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c362bd01a59b4bcf85eeb59713e58b64" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:28.334775 30276 raft_consensus.cc:3037] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.335636 30276 raft_consensus.cc:2454] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:28.335636 30407 raft_consensus.cc:2454] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:28.335919 30079 leader_election.cc:302] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:28.336105 31061 raft_consensus.cc:2781] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:28.336175 31061 raft_consensus.cc:686] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:28.336731 31061 consensus_queue.cc:237] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:28.338630 29868 catalog_manager.cc:5484] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
W20240501 14:09:28.516502 31958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:28.516722 31958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:28.519886 31958 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
I20240501 14:09:28.571000 31061 consensus_queue.cc:1035] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:28.575701 31718 consensus_queue.cc:1035] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:28.598407 30407 raft_consensus.cc:1261] T 1698ca3ec26741bb9b3131fe67c9682b P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:28.598394 30146 raft_consensus.cc:1261] T 1698ca3ec26741bb9b3131fe67c9682b P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:28.598780 31925 consensus_queue.cc:1035] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:28.599172 31925 consensus_queue.cc:1035] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:28.602782 30276 raft_consensus.cc:1261] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:28.603147 31061 consensus_queue.cc:1035] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:28.604547 30406 raft_consensus.cc:1261] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:28.605129 31718 consensus_queue.cc:1035] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:28.611194 31994 mvcc.cc:205] Tried to move back new op lower bound from 7022889240972574720 to 7022889239197802496. Current Snapshot: MvccSnapshot[applied={T|T < 7022889240972574720}]
INSERT report
rows total: 2000
time total: 112.083 ms
time per row: 0.0560413 ms
I20240501 14:09:28.671133 29869 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:28.676340 29994 tablet_service.cc:1467] Processing CreateTablet for tablet 5da2beba68ba414281025fcee91ca0d0 (DEFAULT_TABLE table=rebalance_test_table_14 [id=a61705ba694e47f3818ac35bcffaee89]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.676658 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5da2beba68ba414281025fcee91ca0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.676725 30387 tablet_service.cc:1467] Processing CreateTablet for tablet 5da2beba68ba414281025fcee91ca0d0 (DEFAULT_TABLE table=rebalance_test_table_14 [id=a61705ba694e47f3818ac35bcffaee89]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.676985 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5da2beba68ba414281025fcee91ca0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.677412 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 855d8906d14e46ebacfd866861d287e7 (DEFAULT_TABLE table=rebalance_test_table_14 [id=a61705ba694e47f3818ac35bcffaee89]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.677412 29995 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7582b1c4c040849ad1507ccbaf0584 (DEFAULT_TABLE table=rebalance_test_table_14 [id=a61705ba694e47f3818ac35bcffaee89]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.677687 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 855d8906d14e46ebacfd866861d287e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.677786 29995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7582b1c4c040849ad1507ccbaf0584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.679150 30126 tablet_service.cc:1467] Processing CreateTablet for tablet 5da2beba68ba414281025fcee91ca0d0 (DEFAULT_TABLE table=rebalance_test_table_14 [id=a61705ba694e47f3818ac35bcffaee89]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.679152 30124 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7582b1c4c040849ad1507ccbaf0584 (DEFAULT_TABLE table=rebalance_test_table_14 [id=a61705ba694e47f3818ac35bcffaee89]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.679448 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5da2beba68ba414281025fcee91ca0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.679791 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7582b1c4c040849ad1507ccbaf0584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.679965 32018 tablet_bootstrap.cc:492] T 5da2beba68ba414281025fcee91ca0d0 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:28.680734 32018 tablet_bootstrap.cc:654] T 5da2beba68ba414281025fcee91ca0d0 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.681620 32020 tablet_bootstrap.cc:492] T 5da2beba68ba414281025fcee91ca0d0 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:28.681811 30256 tablet_service.cc:1467] Processing CreateTablet for tablet 855d8906d14e46ebacfd866861d287e7 (DEFAULT_TABLE table=rebalance_test_table_14 [id=a61705ba694e47f3818ac35bcffaee89]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.681994 30386 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7582b1c4c040849ad1507ccbaf0584 (DEFAULT_TABLE table=rebalance_test_table_14 [id=a61705ba694e47f3818ac35bcffaee89]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.682072 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 855d8906d14e46ebacfd866861d287e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.682217 30386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7582b1c4c040849ad1507ccbaf0584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.682178 30387 tablet_service.cc:1467] Processing CreateTablet for tablet 855d8906d14e46ebacfd866861d287e7 (DEFAULT_TABLE table=rebalance_test_table_14 [id=a61705ba694e47f3818ac35bcffaee89]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.682410 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 855d8906d14e46ebacfd866861d287e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.682662 32020 tablet_bootstrap.cc:654] T 5da2beba68ba414281025fcee91ca0d0 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.683990 32018 tablet_bootstrap.cc:492] T 5da2beba68ba414281025fcee91ca0d0 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:28.684068 32018 ts_tablet_manager.cc:1367] T 5da2beba68ba414281025fcee91ca0d0 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:28.684608 32022 tablet_bootstrap.cc:492] T 855d8906d14e46ebacfd866861d287e7 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:28.684571 32018 raft_consensus.cc:348] T 5da2beba68ba414281025fcee91ca0d0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:28.684679 32018 raft_consensus.cc:374] T 5da2beba68ba414281025fcee91ca0d0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.684725 32018 raft_consensus.cc:729] T 5da2beba68ba414281025fcee91ca0d0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.684846 32018 consensus_queue.cc:260] T 5da2beba68ba414281025fcee91ca0d0 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:28.685169 32017 tablet_bootstrap.cc:492] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:28.685494 32022 tablet_bootstrap.cc:654] T 855d8906d14e46ebacfd866861d287e7 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.685746 32018 ts_tablet_manager.cc:1398] T 5da2beba68ba414281025fcee91ca0d0 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:28.685894 32017 tablet_bootstrap.cc:654] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.685910 32018 tablet_bootstrap.cc:492] T 8a7582b1c4c040849ad1507ccbaf0584 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:28.686892 32017 tablet_bootstrap.cc:492] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:28.686910 32018 tablet_bootstrap.cc:654] T 8a7582b1c4c040849ad1507ccbaf0584 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.687048 32017 ts_tablet_manager.cc:1367] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:28.687458 32017 raft_consensus.cc:348] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:28.687590 32017 raft_consensus.cc:374] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.687649 32017 raft_consensus.cc:729] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.687812 32017 consensus_queue.cc:260] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:28.688071 32017 ts_tablet_manager.cc:1398] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:28.688297 32017 tablet_bootstrap.cc:492] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:28.688627 32020 tablet_bootstrap.cc:492] T 5da2beba68ba414281025fcee91ca0d0 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:28.688715 32020 ts_tablet_manager.cc:1367] T 5da2beba68ba414281025fcee91ca0d0 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:28.688851 32022 tablet_bootstrap.cc:492] T 855d8906d14e46ebacfd866861d287e7 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:28.689000 32022 ts_tablet_manager.cc:1367] T 855d8906d14e46ebacfd866861d287e7 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:28.689184 32020 raft_consensus.cc:348] T 5da2beba68ba414281025fcee91ca0d0 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:28.689252 32017 tablet_bootstrap.cc:654] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.689296 32020 raft_consensus.cc:374] T 5da2beba68ba414281025fcee91ca0d0 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.689334 32020 raft_consensus.cc:729] T 5da2beba68ba414281025fcee91ca0d0 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.689539 32020 consensus_queue.cc:260] T 5da2beba68ba414281025fcee91ca0d0 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:28.689617 32022 raft_consensus.cc:348] T 855d8906d14e46ebacfd866861d287e7 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:28.689826 32022 raft_consensus.cc:374] T 855d8906d14e46ebacfd866861d287e7 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.689903 32022 raft_consensus.cc:729] T 855d8906d14e46ebacfd866861d287e7 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.690073 32022 consensus_queue.cc:260] T 855d8906d14e46ebacfd866861d287e7 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:28.690599 32018 tablet_bootstrap.cc:492] T 8a7582b1c4c040849ad1507ccbaf0584 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:28.690673 32018 ts_tablet_manager.cc:1367] T 8a7582b1c4c040849ad1507ccbaf0584 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:28.691088 32018 raft_consensus.cc:348] T 8a7582b1c4c040849ad1507ccbaf0584 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:28.691179 32018 raft_consensus.cc:374] T 8a7582b1c4c040849ad1507ccbaf0584 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.691216 32018 raft_consensus.cc:729] T 8a7582b1c4c040849ad1507ccbaf0584 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.691335 32018 consensus_queue.cc:260] T 8a7582b1c4c040849ad1507ccbaf0584 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:28.691509 32017 tablet_bootstrap.cc:492] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:28.691622 32017 ts_tablet_manager.cc:1367] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:28.691644 32018 ts_tablet_manager.cc:1398] T 8a7582b1c4c040849ad1507ccbaf0584 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:28.691802 32018 tablet_bootstrap.cc:492] T 855d8906d14e46ebacfd866861d287e7 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:28.691812 32020 ts_tablet_manager.cc:1398] T 5da2beba68ba414281025fcee91ca0d0 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:28.692128 32017 raft_consensus.cc:348] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:28.692210 32017 raft_consensus.cc:374] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.692265 32017 raft_consensus.cc:729] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.692370 32017 consensus_queue.cc:260] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:28.692682 32020 tablet_bootstrap.cc:492] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:28.693403 32017 ts_tablet_manager.cc:1398] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:28.693563 32017 tablet_bootstrap.cc:492] T 8a7582b1c4c040849ad1507ccbaf0584 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:28.694388 32017 tablet_bootstrap.cc:654] T 8a7582b1c4c040849ad1507ccbaf0584 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.694530 32020 tablet_bootstrap.cc:654] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.695326 32018 tablet_bootstrap.cc:654] T 855d8906d14e46ebacfd866861d287e7 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.696048 32022 ts_tablet_manager.cc:1398] T 855d8906d14e46ebacfd866861d287e7 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:28.696507 32018 tablet_bootstrap.cc:492] T 855d8906d14e46ebacfd866861d287e7 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:28.696583 32018 ts_tablet_manager.cc:1367] T 855d8906d14e46ebacfd866861d287e7 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:09:28.696933 32017 tablet_bootstrap.cc:492] T 8a7582b1c4c040849ad1507ccbaf0584 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:28.697021 32017 ts_tablet_manager.cc:1367] T 8a7582b1c4c040849ad1507ccbaf0584 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:28.697181 32018 raft_consensus.cc:348] T 855d8906d14e46ebacfd866861d287e7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:28.697302 32018 raft_consensus.cc:374] T 855d8906d14e46ebacfd866861d287e7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.697340 32018 raft_consensus.cc:729] T 855d8906d14e46ebacfd866861d287e7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.697467 32018 consensus_queue.cc:260] T 855d8906d14e46ebacfd866861d287e7 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:28.697803 32020 tablet_bootstrap.cc:492] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:28.697878 32020 ts_tablet_manager.cc:1367] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 14:09:28.697901 32018 ts_tablet_manager.cc:1398] T 855d8906d14e46ebacfd866861d287e7 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:28.698312 32020 raft_consensus.cc:348] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:28.698427 32020 raft_consensus.cc:374] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.698469 32020 raft_consensus.cc:729] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.698582 32020 consensus_queue.cc:260] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:28.698897 32020 ts_tablet_manager.cc:1398] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:28.697365 32017 raft_consensus.cc:348] T 8a7582b1c4c040849ad1507ccbaf0584 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:28.699041 32017 raft_consensus.cc:374] T 8a7582b1c4c040849ad1507ccbaf0584 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.699092 32017 raft_consensus.cc:729] T 8a7582b1c4c040849ad1507ccbaf0584 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.699224 32017 consensus_queue.cc:260] T 8a7582b1c4c040849ad1507ccbaf0584 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:28.699579 32017 ts_tablet_manager.cc:1398] T 8a7582b1c4c040849ad1507ccbaf0584 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:28.719132 31858 raft_consensus.cc:482] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:28.719244 31858 raft_consensus.cc:504] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:28.719614 31858 leader_election.cc:288] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:28.719883 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855d8906d14e46ebacfd866861d287e7" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:28.719947 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855d8906d14e46ebacfd866861d287e7" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:28.720002 30407 raft_consensus.cc:2454] T 855d8906d14e46ebacfd866861d287e7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:28.720089 30276 raft_consensus.cc:2454] T 855d8906d14e46ebacfd866861d287e7 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:28.720315 29948 leader_election.cc:302] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:28.720487 31858 raft_consensus.cc:2781] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:28.720539 31858 raft_consensus.cc:482] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:28.720594 31858 raft_consensus.cc:3037] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.721418 31858 raft_consensus.cc:504] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:28.721727 31858 leader_election.cc:288] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:28.721971 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855d8906d14e46ebacfd866861d287e7" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:28.722024 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855d8906d14e46ebacfd866861d287e7" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:28.722090 30407 raft_consensus.cc:3037] T 855d8906d14e46ebacfd866861d287e7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.722116 30276 raft_consensus.cc:3037] T 855d8906d14e46ebacfd866861d287e7 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.723006 30407 raft_consensus.cc:2454] T 855d8906d14e46ebacfd866861d287e7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:28.723006 30276 raft_consensus.cc:2454] T 855d8906d14e46ebacfd866861d287e7 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:28.723281 29948 leader_election.cc:302] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:28.723425 31858 raft_consensus.cc:2781] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:28.723513 31858 raft_consensus.cc:686] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:28.723644 31858 consensus_queue.cc:237] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:28.725397 29866 catalog_manager.cc:5484] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:28.801165 31858 raft_consensus.cc:482] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:28.801276 31858 raft_consensus.cc:504] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:28.801692 31858 leader_election.cc:288] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:28.802067 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da2beba68ba414281025fcee91ca0d0" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:28.802234 30407 raft_consensus.cc:2454] T 5da2beba68ba414281025fcee91ca0d0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:28.802284 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da2beba68ba414281025fcee91ca0d0" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:28.802407 30146 raft_consensus.cc:2454] T 5da2beba68ba414281025fcee91ca0d0 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:28.802485 29948 leader_election.cc:302] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:28.802665 31787 raft_consensus.cc:2781] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:28.802717 31787 raft_consensus.cc:482] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:28.802778 31787 raft_consensus.cc:3037] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.803718 31787 raft_consensus.cc:504] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:28.804001 31787 leader_election.cc:288] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:28.804313 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da2beba68ba414281025fcee91ca0d0" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:28.804316 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da2beba68ba414281025fcee91ca0d0" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:28.804404 30146 raft_consensus.cc:3037] T 5da2beba68ba414281025fcee91ca0d0 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.804409 30407 raft_consensus.cc:3037] T 5da2beba68ba414281025fcee91ca0d0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.805137 30407 raft_consensus.cc:2454] T 5da2beba68ba414281025fcee91ca0d0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:28.805292 30146 raft_consensus.cc:2454] T 5da2beba68ba414281025fcee91ca0d0 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:28.805574 29948 leader_election.cc:302] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:28.805784 31787 raft_consensus.cc:2781] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:28.805850 31787 raft_consensus.cc:686] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:28.805964 31787 consensus_queue.cc:237] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:28.807825 29866 catalog_manager.cc:5484] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:28.814199 32001 raft_consensus.cc:482] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:28.814285 32001 raft_consensus.cc:504] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:28.814833 32001 leader_election.cc:288] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:28.815148 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7582b1c4c040849ad1507ccbaf0584" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:28.815269 30407 raft_consensus.cc:2454] T 8a7582b1c4c040849ad1507ccbaf0584 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:28.815558 30079 leader_election.cc:302] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:28.815748 32001 raft_consensus.cc:2781] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:28.815816 32001 raft_consensus.cc:482] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:28.815856 32001 raft_consensus.cc:3037] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.816143 30015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7582b1c4c040849ad1507ccbaf0584" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:28.816289 30015 raft_consensus.cc:2454] T 8a7582b1c4c040849ad1507ccbaf0584 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:28.816866 32001 raft_consensus.cc:504] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:28.817174 32001 leader_election.cc:288] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:28.817412 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7582b1c4c040849ad1507ccbaf0584" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:28.817463 30015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7582b1c4c040849ad1507ccbaf0584" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:28.817514 30407 raft_consensus.cc:3037] T 8a7582b1c4c040849ad1507ccbaf0584 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.817555 30015 raft_consensus.cc:3037] T 8a7582b1c4c040849ad1507ccbaf0584 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.818356 30407 raft_consensus.cc:2454] T 8a7582b1c4c040849ad1507ccbaf0584 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:28.818521 30015 raft_consensus.cc:2454] T 8a7582b1c4c040849ad1507ccbaf0584 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:28.818641 30079 leader_election.cc:302] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:28.818789 32001 raft_consensus.cc:2781] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:28.818878 32001 raft_consensus.cc:686] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:28.819016 32001 consensus_queue.cc:237] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:28.820662 29866 catalog_manager.cc:5484] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
W20240501 14:09:28.938745 32033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:28.938978 32033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:28.942138 32033 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240501 14:09:29.011942 30407 raft_consensus.cc:1261] T 855d8906d14e46ebacfd866861d287e7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.011929 30276 raft_consensus.cc:1261] T 855d8906d14e46ebacfd866861d287e7 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.012727 31858 consensus_queue.cc:1035] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.013269 30146 raft_consensus.cc:1261] T 5da2beba68ba414281025fcee91ca0d0 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.013433 30407 raft_consensus.cc:1261] T 5da2beba68ba414281025fcee91ca0d0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.013916 31787 consensus_queue.cc:1035] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.014235 31787 consensus_queue.cc:1035] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.014505 31787 consensus_queue.cc:1035] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.024183 31983 mvcc.cc:205] Tried to move back new op lower bound from 7022889242668052480 to 7022889241493233664. Current Snapshot: MvccSnapshot[applied={T|T < 7022889242668052480}]
I20240501 14:09:29.026386 30407 raft_consensus.cc:1261] T 8a7582b1c4c040849ad1507ccbaf0584 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.027714 30015 raft_consensus.cc:1261] T 8a7582b1c4c040849ad1507ccbaf0584 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.027802 32001 consensus_queue.cc:1035] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.028494 32001 consensus_queue.cc:1035] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 99.7643 ms
time per row: 0.0498821 ms
I20240501 14:09:29.083299 29867 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:29.088752 30124 tablet_service.cc:1467] Processing CreateTablet for tablet 15c98173a71d4f73b8f6af24ad01d7c4 (DEFAULT_TABLE table=rebalance_test_table_15 [id=df0504ee71e643e3a5ec9a20e1037981]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.088752 30256 tablet_service.cc:1467] Processing CreateTablet for tablet 15c98173a71d4f73b8f6af24ad01d7c4 (DEFAULT_TABLE table=rebalance_test_table_15 [id=df0504ee71e643e3a5ec9a20e1037981]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.089030 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15c98173a71d4f73b8f6af24ad01d7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.089030 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15c98173a71d4f73b8f6af24ad01d7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.089669 30386 tablet_service.cc:1467] Processing CreateTablet for tablet 371c725533d84656b7a76c1bc981d038 (DEFAULT_TABLE table=rebalance_test_table_15 [id=df0504ee71e643e3a5ec9a20e1037981]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.089937 30386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371c725533d84656b7a76c1bc981d038. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.091101 32020 tablet_bootstrap.cc:492] T 15c98173a71d4f73b8f6af24ad01d7c4 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:29.091260 30387 tablet_service.cc:1467] Processing CreateTablet for tablet deee2ab875274d4eaeab84e698241ca8 (DEFAULT_TABLE table=rebalance_test_table_15 [id=df0504ee71e643e3a5ec9a20e1037981]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.091260 30126 tablet_service.cc:1467] Processing CreateTablet for tablet 371c725533d84656b7a76c1bc981d038 (DEFAULT_TABLE table=rebalance_test_table_15 [id=df0504ee71e643e3a5ec9a20e1037981]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.091471 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371c725533d84656b7a76c1bc981d038. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.091471 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deee2ab875274d4eaeab84e698241ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.092015 32020 tablet_bootstrap.cc:654] T 15c98173a71d4f73b8f6af24ad01d7c4 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.092619 32018 tablet_bootstrap.cc:492] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:29.093289 32018 tablet_bootstrap.cc:654] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.093328 32022 tablet_bootstrap.cc:492] T 15c98173a71d4f73b8f6af24ad01d7c4 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:29.093997 32022 tablet_bootstrap.cc:654] T 15c98173a71d4f73b8f6af24ad01d7c4 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.094987 32018 tablet_bootstrap.cc:492] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:29.094990 29995 tablet_service.cc:1467] Processing CreateTablet for tablet 15c98173a71d4f73b8f6af24ad01d7c4 (DEFAULT_TABLE table=rebalance_test_table_15 [id=df0504ee71e643e3a5ec9a20e1037981]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.095068 32018 ts_tablet_manager.cc:1367] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:29.095208 29995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15c98173a71d4f73b8f6af24ad01d7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.095429 32018 raft_consensus.cc:348] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:29.095513 32018 raft_consensus.cc:374] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.095546 32018 raft_consensus.cc:729] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.095640 32018 consensus_queue.cc:260] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:29.095873 32018 ts_tablet_manager.cc:1398] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:29.096040 32018 tablet_bootstrap.cc:492] T deee2ab875274d4eaeab84e698241ca8 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:29.096441 29993 tablet_service.cc:1467] Processing CreateTablet for tablet deee2ab875274d4eaeab84e698241ca8 (DEFAULT_TABLE table=rebalance_test_table_15 [id=df0504ee71e643e3a5ec9a20e1037981]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.096608 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deee2ab875274d4eaeab84e698241ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.096858 32018 tablet_bootstrap.cc:654] T deee2ab875274d4eaeab84e698241ca8 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.097764 32020 tablet_bootstrap.cc:492] T 15c98173a71d4f73b8f6af24ad01d7c4 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:29.097842 32020 ts_tablet_manager.cc:1367] T 15c98173a71d4f73b8f6af24ad01d7c4 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240501 14:09:29.098233 32020 raft_consensus.cc:348] T 15c98173a71d4f73b8f6af24ad01d7c4 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:29.098320 32020 raft_consensus.cc:374] T 15c98173a71d4f73b8f6af24ad01d7c4 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.098349 32020 raft_consensus.cc:729] T 15c98173a71d4f73b8f6af24ad01d7c4 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.098448 32017 tablet_bootstrap.cc:492] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:29.098467 32020 consensus_queue.cc:260] T 15c98173a71d4f73b8f6af24ad01d7c4 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:29.098752 32020 ts_tablet_manager.cc:1398] T 15c98173a71d4f73b8f6af24ad01d7c4 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.098914 32020 tablet_bootstrap.cc:492] T 371c725533d84656b7a76c1bc981d038 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:29.098927 30256 tablet_service.cc:1467] Processing CreateTablet for tablet deee2ab875274d4eaeab84e698241ca8 (DEFAULT_TABLE table=rebalance_test_table_15 [id=df0504ee71e643e3a5ec9a20e1037981]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.098927 30257 tablet_service.cc:1467] Processing CreateTablet for tablet 371c725533d84656b7a76c1bc981d038 (DEFAULT_TABLE table=rebalance_test_table_15 [id=df0504ee71e643e3a5ec9a20e1037981]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.099205 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deee2ab875274d4eaeab84e698241ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.099334 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371c725533d84656b7a76c1bc981d038. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.099341 32017 tablet_bootstrap.cc:654] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.100701 32020 tablet_bootstrap.cc:654] T 371c725533d84656b7a76c1bc981d038 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.101135 32018 tablet_bootstrap.cc:492] T deee2ab875274d4eaeab84e698241ca8 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:29.101689 32018 ts_tablet_manager.cc:1367] T deee2ab875274d4eaeab84e698241ca8 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:29.102208 32020 tablet_bootstrap.cc:492] T 371c725533d84656b7a76c1bc981d038 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:29.102294 32020 ts_tablet_manager.cc:1367] T 371c725533d84656b7a76c1bc981d038 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:29.102216 32018 raft_consensus.cc:348] T deee2ab875274d4eaeab84e698241ca8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:29.102449 32018 raft_consensus.cc:374] T deee2ab875274d4eaeab84e698241ca8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.102537 32018 raft_consensus.cc:729] T deee2ab875274d4eaeab84e698241ca8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.102705 32018 consensus_queue.cc:260] T deee2ab875274d4eaeab84e698241ca8 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:29.102716 32020 raft_consensus.cc:348] T 371c725533d84656b7a76c1bc981d038 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:29.102819 32020 raft_consensus.cc:374] T 371c725533d84656b7a76c1bc981d038 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.102867 32020 raft_consensus.cc:729] T 371c725533d84656b7a76c1bc981d038 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.102995 32020 consensus_queue.cc:260] T 371c725533d84656b7a76c1bc981d038 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:29.103076 32018 ts_tablet_manager.cc:1398] T deee2ab875274d4eaeab84e698241ca8 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.103284 32020 ts_tablet_manager.cc:1398] T 371c725533d84656b7a76c1bc981d038 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 14:09:29.104077 32022 tablet_bootstrap.cc:492] T 15c98173a71d4f73b8f6af24ad01d7c4 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:29.104153 32022 ts_tablet_manager.cc:1367] T 15c98173a71d4f73b8f6af24ad01d7c4 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240501 14:09:29.104526 32022 raft_consensus.cc:348] T 15c98173a71d4f73b8f6af24ad01d7c4 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:29.104632 32022 raft_consensus.cc:374] T 15c98173a71d4f73b8f6af24ad01d7c4 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.104663 32022 raft_consensus.cc:729] T 15c98173a71d4f73b8f6af24ad01d7c4 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.104763 32022 consensus_queue.cc:260] T 15c98173a71d4f73b8f6af24ad01d7c4 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:29.105057 32022 ts_tablet_manager.cc:1398] T 15c98173a71d4f73b8f6af24ad01d7c4 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:29.105232 32022 tablet_bootstrap.cc:492] T deee2ab875274d4eaeab84e698241ca8 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:29.105649 32017 tablet_bootstrap.cc:492] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:29.105777 32017 ts_tablet_manager.cc:1367] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:29.106247 32022 tablet_bootstrap.cc:654] T deee2ab875274d4eaeab84e698241ca8 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.106261 32017 raft_consensus.cc:348] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:29.106356 32017 raft_consensus.cc:374] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.106379 32017 raft_consensus.cc:729] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.106468 32017 consensus_queue.cc:260] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:29.106676 32017 ts_tablet_manager.cc:1398] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.106879 32017 tablet_bootstrap.cc:492] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:29.107501 32022 tablet_bootstrap.cc:492] T deee2ab875274d4eaeab84e698241ca8 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:29.107563 32022 ts_tablet_manager.cc:1367] T deee2ab875274d4eaeab84e698241ca8 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:29.107688 32017 tablet_bootstrap.cc:654] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.107955 32022 raft_consensus.cc:348] T deee2ab875274d4eaeab84e698241ca8 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:29.108037 32022 raft_consensus.cc:374] T deee2ab875274d4eaeab84e698241ca8 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.108058 32022 raft_consensus.cc:729] T deee2ab875274d4eaeab84e698241ca8 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.108157 32022 consensus_queue.cc:260] T deee2ab875274d4eaeab84e698241ca8 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:29.108454 32022 ts_tablet_manager.cc:1398] T deee2ab875274d4eaeab84e698241ca8 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.108603 32022 tablet_bootstrap.cc:492] T 371c725533d84656b7a76c1bc981d038 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:29.108633 32017 tablet_bootstrap.cc:492] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:29.108714 32017 ts_tablet_manager.cc:1367] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:29.109233 32017 raft_consensus.cc:348] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:29.109313 32017 raft_consensus.cc:374] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.109347 32017 raft_consensus.cc:729] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.109490 32022 tablet_bootstrap.cc:654] T 371c725533d84656b7a76c1bc981d038 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.109493 32017 consensus_queue.cc:260] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:29.109815 32017 ts_tablet_manager.cc:1398] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.110659 32022 tablet_bootstrap.cc:492] T 371c725533d84656b7a76c1bc981d038 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:29.111321 32022 ts_tablet_manager.cc:1367] T 371c725533d84656b7a76c1bc981d038 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:29.111729 32022 raft_consensus.cc:348] T 371c725533d84656b7a76c1bc981d038 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:29.111832 32022 raft_consensus.cc:374] T 371c725533d84656b7a76c1bc981d038 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.111862 32022 raft_consensus.cc:729] T 371c725533d84656b7a76c1bc981d038 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.112011 32022 consensus_queue.cc:260] T 371c725533d84656b7a76c1bc981d038 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:29.112429 32022 ts_tablet_manager.cc:1398] T 371c725533d84656b7a76c1bc981d038 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.157145 31858 raft_consensus.cc:482] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:29.157245 31858 raft_consensus.cc:504] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:29.157589 31858 leader_election.cc:288] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:29.157845 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c98173a71d4f73b8f6af24ad01d7c4" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:29.157912 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c98173a71d4f73b8f6af24ad01d7c4" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:29.157969 30276 raft_consensus.cc:2454] T 15c98173a71d4f73b8f6af24ad01d7c4 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:29.158046 30146 raft_consensus.cc:2454] T 15c98173a71d4f73b8f6af24ad01d7c4 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:29.158246 29950 leader_election.cc:302] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:29.158437 31858 raft_consensus.cc:2781] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:29.158505 31858 raft_consensus.cc:482] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:29.158551 31858 raft_consensus.cc:3037] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.159621 31858 raft_consensus.cc:504] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:29.159952 31858 leader_election.cc:288] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:29.160207 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c98173a71d4f73b8f6af24ad01d7c4" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:29.160337 30276 raft_consensus.cc:3037] T 15c98173a71d4f73b8f6af24ad01d7c4 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.160310 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c98173a71d4f73b8f6af24ad01d7c4" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:29.160423 30146 raft_consensus.cc:3037] T 15c98173a71d4f73b8f6af24ad01d7c4 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.161125 30276 raft_consensus.cc:2454] T 15c98173a71d4f73b8f6af24ad01d7c4 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:29.161129 30146 raft_consensus.cc:2454] T 15c98173a71d4f73b8f6af24ad01d7c4 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:29.161365 29951 leader_election.cc:302] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:29.161535 31858 raft_consensus.cc:2781] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:29.161626 31858 raft_consensus.cc:686] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:29.161756 31858 consensus_queue.cc:237] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:29.163336 29867 catalog_manager.cc:5484] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:29.256999 31858 raft_consensus.cc:482] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:29.257133 31858 raft_consensus.cc:504] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:29.257524 31858 leader_election.cc:288] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:29.257813 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deee2ab875274d4eaeab84e698241ca8" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:29.257838 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deee2ab875274d4eaeab84e698241ca8" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:29.257943 30277 raft_consensus.cc:2454] T deee2ab875274d4eaeab84e698241ca8 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:29.257941 30406 raft_consensus.cc:2454] T deee2ab875274d4eaeab84e698241ca8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:29.258232 29950 leader_election.cc:302] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:29.258388 31787 raft_consensus.cc:2781] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:29.258447 31787 raft_consensus.cc:482] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:29.258484 31787 raft_consensus.cc:3037] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.259308 31787 raft_consensus.cc:504] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:29.259600 31787 leader_election.cc:288] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:29.259872 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deee2ab875274d4eaeab84e698241ca8" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:29.259889 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deee2ab875274d4eaeab84e698241ca8" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:29.259958 30406 raft_consensus.cc:3037] T deee2ab875274d4eaeab84e698241ca8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.259980 30277 raft_consensus.cc:3037] T deee2ab875274d4eaeab84e698241ca8 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.260836 30277 raft_consensus.cc:2454] T deee2ab875274d4eaeab84e698241ca8 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:29.260866 30406 raft_consensus.cc:2454] T deee2ab875274d4eaeab84e698241ca8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:29.261154 29948 leader_election.cc:302] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:29.261448 31787 raft_consensus.cc:2781] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:29.261621 31787 raft_consensus.cc:686] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:29.261746 31787 consensus_queue.cc:237] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:29.263471 29867 catalog_manager.cc:5484] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:29.304373 31874 raft_consensus.cc:482] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:29.304486 31874 raft_consensus.cc:504] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:29.304826 31874 leader_election.cc:288] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:29.305083 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371c725533d84656b7a76c1bc981d038" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:29.305124 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371c725533d84656b7a76c1bc981d038" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:29.305207 30277 raft_consensus.cc:2454] T 371c725533d84656b7a76c1bc981d038 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:29.305236 30146 raft_consensus.cc:2454] T 371c725533d84656b7a76c1bc981d038 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:29.305452 30343 leader_election.cc:302] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:29.305627 31874 raft_consensus.cc:2781] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:29.305677 31874 raft_consensus.cc:482] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:29.305704 31874 raft_consensus.cc:3037] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.306478 31874 raft_consensus.cc:504] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:29.306753 31874 leader_election.cc:288] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:29.307003 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371c725533d84656b7a76c1bc981d038" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:29.307054 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371c725533d84656b7a76c1bc981d038" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:29.307111 30277 raft_consensus.cc:3037] T 371c725533d84656b7a76c1bc981d038 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.307161 30146 raft_consensus.cc:3037] T 371c725533d84656b7a76c1bc981d038 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.308068 30146 raft_consensus.cc:2454] T 371c725533d84656b7a76c1bc981d038 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:29.308068 30277 raft_consensus.cc:2454] T 371c725533d84656b7a76c1bc981d038 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:29.308331 30343 leader_election.cc:302] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:29.308480 31874 raft_consensus.cc:2781] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:29.308553 31874 raft_consensus.cc:686] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:29.308682 31874 consensus_queue.cc:237] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:29.310452 29867 catalog_manager.cc:5484] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
W20240501 14:09:29.441202 32083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:29.441460 32083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:29.445297 32083 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
I20240501 14:09:29.527415 30146 raft_consensus.cc:1261] T 15c98173a71d4f73b8f6af24ad01d7c4 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.527626 30406 raft_consensus.cc:1261] T deee2ab875274d4eaeab84e698241ca8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.528064 32080 consensus_queue.cc:1035] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.528071 31787 consensus_queue.cc:1035] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.528492 30277 raft_consensus.cc:1261] T deee2ab875274d4eaeab84e698241ca8 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.529233 31858 consensus_queue.cc:1035] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.530158 30145 raft_consensus.cc:1261] T 371c725533d84656b7a76c1bc981d038 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.531319 30276 raft_consensus.cc:1261] T 15c98173a71d4f73b8f6af24ad01d7c4 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.531864 30277 raft_consensus.cc:1261] T 371c725533d84656b7a76c1bc981d038 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.532215 31874 consensus_queue.cc:1035] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.532418 32069 consensus_queue.cc:1035] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.535579 31787 consensus_queue.cc:1035] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.549014 32003 mvcc.cc:205] Tried to move back new op lower bound from 7022889244783906816 to 7022889243889668096. Current Snapshot: MvccSnapshot[applied={T|T < 7022889244783906816}]
I20240501 14:09:29.560863 32118 mvcc.cc:205] Tried to move back new op lower bound from 7022889244783906816 to 7022889243889668096. Current Snapshot: MvccSnapshot[applied={T|T < 7022889244783906816}]
INSERT report
rows total: 2000
time total: 115.815 ms
time per row: 0.0579073 ms
I20240501 14:09:29.602022 29869 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:29.607335 29995 tablet_service.cc:1467] Processing CreateTablet for tablet 796cef2485c94b76a5f3e1944b73dcf6 (DEFAULT_TABLE table=rebalance_test_table_16 [id=86d7db413f934f379c4369701c5c13cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.607653 29995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 796cef2485c94b76a5f3e1944b73dcf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.607620 30387 tablet_service.cc:1467] Processing CreateTablet for tablet 464516e077954ac98413609f05665183 (DEFAULT_TABLE table=rebalance_test_table_16 [id=86d7db413f934f379c4369701c5c13cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.607846 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 464516e077954ac98413609f05665183. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.608088 30126 tablet_service.cc:1467] Processing CreateTablet for tablet 796cef2485c94b76a5f3e1944b73dcf6 (DEFAULT_TABLE table=rebalance_test_table_16 [id=86d7db413f934f379c4369701c5c13cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.608341 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 796cef2485c94b76a5f3e1944b73dcf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.610078 32017 tablet_bootstrap.cc:492] T 796cef2485c94b76a5f3e1944b73dcf6 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:29.610466 30124 tablet_service.cc:1467] Processing CreateTablet for tablet 464516e077954ac98413609f05665183 (DEFAULT_TABLE table=rebalance_test_table_16 [id=86d7db413f934f379c4369701c5c13cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.610675 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 464516e077954ac98413609f05665183. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.610860 32017 tablet_bootstrap.cc:654] T 796cef2485c94b76a5f3e1944b73dcf6 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.611744 32017 tablet_bootstrap.cc:492] T 796cef2485c94b76a5f3e1944b73dcf6 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:29.611804 32017 ts_tablet_manager.cc:1367] T 796cef2485c94b76a5f3e1944b73dcf6 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:29.612479 32017 raft_consensus.cc:348] T 796cef2485c94b76a5f3e1944b73dcf6 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:29.612555 32017 raft_consensus.cc:374] T 796cef2485c94b76a5f3e1944b73dcf6 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.612576 32017 raft_consensus.cc:729] T 796cef2485c94b76a5f3e1944b73dcf6 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.612560 30386 tablet_service.cc:1467] Processing CreateTablet for tablet ffcce6d1868242d69287d5831f4233f9 (DEFAULT_TABLE table=rebalance_test_table_16 [id=86d7db413f934f379c4369701c5c13cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.612670 32017 consensus_queue.cc:260] T 796cef2485c94b76a5f3e1944b73dcf6 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:29.612761 30386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffcce6d1868242d69287d5831f4233f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.612903 32017 ts_tablet_manager.cc:1398] T 796cef2485c94b76a5f3e1944b73dcf6 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.614936 30257 tablet_service.cc:1467] Processing CreateTablet for tablet 796cef2485c94b76a5f3e1944b73dcf6 (DEFAULT_TABLE table=rebalance_test_table_16 [id=86d7db413f934f379c4369701c5c13cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.614990 29993 tablet_service.cc:1467] Processing CreateTablet for tablet ffcce6d1868242d69287d5831f4233f9 (DEFAULT_TABLE table=rebalance_test_table_16 [id=86d7db413f934f379c4369701c5c13cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.615178 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffcce6d1868242d69287d5831f4233f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.615180 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 796cef2485c94b76a5f3e1944b73dcf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.615268 30256 tablet_service.cc:1467] Processing CreateTablet for tablet ffcce6d1868242d69287d5831f4233f9 (DEFAULT_TABLE table=rebalance_test_table_16 [id=86d7db413f934f379c4369701c5c13cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.615509 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffcce6d1868242d69287d5831f4233f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.617376 29995 tablet_service.cc:1467] Processing CreateTablet for tablet 464516e077954ac98413609f05665183 (DEFAULT_TABLE table=rebalance_test_table_16 [id=86d7db413f934f379c4369701c5c13cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.617591 29995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 464516e077954ac98413609f05665183. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.617687 32017 tablet_bootstrap.cc:492] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:29.617990 32123 tablet_bootstrap.cc:492] T 464516e077954ac98413609f05665183 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:29.618539 32017 tablet_bootstrap.cc:654] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.620179 32017 tablet_bootstrap.cc:492] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:29.620332 32017 ts_tablet_manager.cc:1367] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:29.620532 32123 tablet_bootstrap.cc:654] T 464516e077954ac98413609f05665183 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.620756 32124 tablet_bootstrap.cc:492] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:29.620743 32017 raft_consensus.cc:348] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:29.620842 32017 raft_consensus.cc:374] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.620883 32017 raft_consensus.cc:729] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.620993 32017 consensus_queue.cc:260] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:29.621229 32017 ts_tablet_manager.cc:1398] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.621366 32017 tablet_bootstrap.cc:492] T 464516e077954ac98413609f05665183 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:29.621543 32123 tablet_bootstrap.cc:492] T 464516e077954ac98413609f05665183 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:29.621618 32123 ts_tablet_manager.cc:1367] T 464516e077954ac98413609f05665183 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:29.621753 32124 tablet_bootstrap.cc:654] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.622049 32123 raft_consensus.cc:348] T 464516e077954ac98413609f05665183 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:29.622136 32123 raft_consensus.cc:374] T 464516e077954ac98413609f05665183 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.622169 32017 tablet_bootstrap.cc:654] T 464516e077954ac98413609f05665183 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.622179 32123 raft_consensus.cc:729] T 464516e077954ac98413609f05665183 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.622294 32123 consensus_queue.cc:260] T 464516e077954ac98413609f05665183 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:29.622552 32123 ts_tablet_manager.cc:1398] T 464516e077954ac98413609f05665183 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.624063 32123 tablet_bootstrap.cc:492] T ffcce6d1868242d69287d5831f4233f9 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:29.624881 32123 tablet_bootstrap.cc:654] T ffcce6d1868242d69287d5831f4233f9 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.626582 32126 tablet_bootstrap.cc:492] T ffcce6d1868242d69287d5831f4233f9 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:29.627449 32126 tablet_bootstrap.cc:654] T ffcce6d1868242d69287d5831f4233f9 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.629210 32126 tablet_bootstrap.cc:492] T ffcce6d1868242d69287d5831f4233f9 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:29.629221 32017 tablet_bootstrap.cc:492] T 464516e077954ac98413609f05665183 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:29.629297 32017 ts_tablet_manager.cc:1367] T 464516e077954ac98413609f05665183 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:09:29.629415 32124 tablet_bootstrap.cc:492] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:29.629508 32124 ts_tablet_manager.cc:1367] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20240501 14:09:29.629608 32123 tablet_bootstrap.cc:492] T ffcce6d1868242d69287d5831f4233f9 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:29.629691 32123 ts_tablet_manager.cc:1367] T ffcce6d1868242d69287d5831f4233f9 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:29.629648 32017 raft_consensus.cc:348] T 464516e077954ac98413609f05665183 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:29.629738 32017 raft_consensus.cc:374] T 464516e077954ac98413609f05665183 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.629776 32017 raft_consensus.cc:729] T 464516e077954ac98413609f05665183 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.629887 32017 consensus_queue.cc:260] T 464516e077954ac98413609f05665183 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:29.629977 32124 raft_consensus.cc:348] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:29.630115 32124 raft_consensus.cc:374] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.630153 32124 raft_consensus.cc:729] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.630182 32017 ts_tablet_manager.cc:1398] T 464516e077954ac98413609f05665183 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.630295 32124 consensus_queue.cc:260] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:29.630530 32123 raft_consensus.cc:348] T ffcce6d1868242d69287d5831f4233f9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:29.630630 32123 raft_consensus.cc:374] T ffcce6d1868242d69287d5831f4233f9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.630667 32123 raft_consensus.cc:729] T ffcce6d1868242d69287d5831f4233f9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.630774 32123 consensus_queue.cc:260] T ffcce6d1868242d69287d5831f4233f9 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:29.631043 32123 ts_tablet_manager.cc:1398] T ffcce6d1868242d69287d5831f4233f9 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.631743 32124 ts_tablet_manager.cc:1398] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:29.631913 32124 tablet_bootstrap.cc:492] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:29.632678 32126 ts_tablet_manager.cc:1367] T ffcce6d1868242d69287d5831f4233f9 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:29.633158 32126 raft_consensus.cc:348] T ffcce6d1868242d69287d5831f4233f9 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:29.633270 32126 raft_consensus.cc:374] T ffcce6d1868242d69287d5831f4233f9 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.633359 32126 raft_consensus.cc:729] T ffcce6d1868242d69287d5831f4233f9 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.633560 32126 consensus_queue.cc:260] T ffcce6d1868242d69287d5831f4233f9 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:29.633950 32126 ts_tablet_manager.cc:1398] T ffcce6d1868242d69287d5831f4233f9 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.634281 32126 tablet_bootstrap.cc:492] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:29.634826 32124 tablet_bootstrap.cc:654] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.635064 32126 tablet_bootstrap.cc:654] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.636567 32124 tablet_bootstrap.cc:492] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:29.636639 32124 ts_tablet_manager.cc:1367] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:29.636970 32124 raft_consensus.cc:348] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:29.637051 32124 raft_consensus.cc:374] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.637082 32124 raft_consensus.cc:729] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.640326 32124 consensus_queue.cc:260] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:29.640633 32124 ts_tablet_manager.cc:1398] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:29.641319 32126 tablet_bootstrap.cc:492] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:29.641395 32126 ts_tablet_manager.cc:1367] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:29.641716 32126 raft_consensus.cc:348] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:29.641800 32126 raft_consensus.cc:374] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.641826 32126 raft_consensus.cc:729] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.641968 32126 consensus_queue.cc:260] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:29.643354 32126 ts_tablet_manager.cc:1398] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:29.646421 31061 raft_consensus.cc:482] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:29.646504 31061 raft_consensus.cc:504] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:29.646786 31061 leader_election.cc:288] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:29.647053 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464516e077954ac98413609f05665183" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:29.647064 30015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464516e077954ac98413609f05665183" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:29.647151 30407 raft_consensus.cc:2454] T 464516e077954ac98413609f05665183 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:29.647171 30015 raft_consensus.cc:2454] T 464516e077954ac98413609f05665183 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:29.647380 30079 leader_election.cc:302] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:29.647570 31061 raft_consensus.cc:2781] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:29.647612 31061 raft_consensus.cc:482] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:29.647662 31061 raft_consensus.cc:3037] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.648581 31061 raft_consensus.cc:504] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:29.648865 31061 leader_election.cc:288] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:29.649106 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464516e077954ac98413609f05665183" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:29.649150 30015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464516e077954ac98413609f05665183" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:29.649206 30407 raft_consensus.cc:3037] T 464516e077954ac98413609f05665183 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.649245 30015 raft_consensus.cc:3037] T 464516e077954ac98413609f05665183 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.650094 30015 raft_consensus.cc:2454] T 464516e077954ac98413609f05665183 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:29.650142 30407 raft_consensus.cc:2454] T 464516e077954ac98413609f05665183 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:29.650343 30081 leader_election.cc:302] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:29.650502 31061 raft_consensus.cc:2781] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:29.650583 31061 raft_consensus.cc:686] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:29.650697 31061 consensus_queue.cc:237] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:29.652402 29869 catalog_manager.cc:5484] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:29.664364 31061 raft_consensus.cc:482] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:29.664443 31061 raft_consensus.cc:504] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:29.664703 31061 leader_election.cc:288] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:29.665005 30015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796cef2485c94b76a5f3e1944b73dcf6" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:29.665127 30015 raft_consensus.cc:2454] T 796cef2485c94b76a5f3e1944b73dcf6 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:29.665102 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796cef2485c94b76a5f3e1944b73dcf6" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:29.665222 30264 raft_consensus.cc:2454] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:29.665431 30081 leader_election.cc:302] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:29.665563 31061 raft_consensus.cc:2781] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:29.665616 31061 raft_consensus.cc:482] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:29.665660 31061 raft_consensus.cc:3037] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.666354 31061 raft_consensus.cc:504] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:29.666615 31061 leader_election.cc:288] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:29.666893 30015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796cef2485c94b76a5f3e1944b73dcf6" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:29.666944 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "796cef2485c94b76a5f3e1944b73dcf6" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:29.667008 30015 raft_consensus.cc:3037] T 796cef2485c94b76a5f3e1944b73dcf6 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.667033 30264 raft_consensus.cc:3037] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.667894 30015 raft_consensus.cc:2454] T 796cef2485c94b76a5f3e1944b73dcf6 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:29.668035 30264 raft_consensus.cc:2454] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:29.668251 30081 leader_election.cc:302] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:29.668411 31061 raft_consensus.cc:2781] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:29.668511 31061 raft_consensus.cc:686] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:29.668637 31061 consensus_queue.cc:237] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:29.670024 29869 catalog_manager.cc:5484] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:29.697064 31787 raft_consensus.cc:482] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:29.697170 31787 raft_consensus.cc:504] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:29.697500 31787 leader_election.cc:288] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:29.697808 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffcce6d1868242d69287d5831f4233f9" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:29.697819 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffcce6d1868242d69287d5831f4233f9" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:29.697947 30407 raft_consensus.cc:2454] T ffcce6d1868242d69287d5831f4233f9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:29.697973 30264 raft_consensus.cc:2454] T ffcce6d1868242d69287d5831f4233f9 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:29.698206 29948 leader_election.cc:302] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:29.698428 31787 raft_consensus.cc:2781] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:29.698482 31787 raft_consensus.cc:482] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:29.698529 31787 raft_consensus.cc:3037] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.699297 31787 raft_consensus.cc:504] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:29.699632 31787 leader_election.cc:288] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:29.699900 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffcce6d1868242d69287d5831f4233f9" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:29.699914 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffcce6d1868242d69287d5831f4233f9" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:29.699982 30264 raft_consensus.cc:3037] T ffcce6d1868242d69287d5831f4233f9 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.700014 30407 raft_consensus.cc:3037] T ffcce6d1868242d69287d5831f4233f9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.700831 30407 raft_consensus.cc:2454] T ffcce6d1868242d69287d5831f4233f9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:29.700836 30264 raft_consensus.cc:2454] T ffcce6d1868242d69287d5831f4233f9 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:29.701076 29950 leader_election.cc:302] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:29.701270 31787 raft_consensus.cc:2781] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:29.701352 31787 raft_consensus.cc:686] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:29.701468 31787 consensus_queue.cc:237] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:29.703119 29869 catalog_manager.cc:5484] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
W20240501 14:09:29.824182 32138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:29.824469 32138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:29.827363 32138 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
I20240501 14:09:29.937541 30407 raft_consensus.cc:1261] T 464516e077954ac98413609f05665183 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.937569 30015 raft_consensus.cc:1261] T 464516e077954ac98413609f05665183 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.937968 32001 consensus_queue.cc:1035] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.938534 32001 consensus_queue.cc:1035] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.939679 30406 raft_consensus.cc:1261] T ffcce6d1868242d69287d5831f4233f9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.942811 30277 raft_consensus.cc:1261] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.943238 32001 consensus_queue.cc:1035] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.944350 32080 consensus_queue.cc:1035] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.944892 30264 raft_consensus.cc:1261] T ffcce6d1868242d69287d5831f4233f9 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.946740 30013 raft_consensus.cc:1261] T 796cef2485c94b76a5f3e1944b73dcf6 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.947183 32080 consensus_queue.cc:1035] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.955421 31061 consensus_queue.cc:1035] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 141.522 ms
time per row: 0.0707608 ms
I20240501 14:09:30.010203 29864 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:30.015497 30257 tablet_service.cc:1467] Processing CreateTablet for tablet 44949ddf1eac45cfa4fc8e1c532cf7bb (DEFAULT_TABLE table=rebalance_test_table_17 [id=9a4b9a0a2fe64ab2a64d6679185c2a33]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.015801 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44949ddf1eac45cfa4fc8e1c532cf7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.016084 30255 tablet_service.cc:1467] Processing CreateTablet for tablet 4a5869fb581b4939ad2207ab9200d674 (DEFAULT_TABLE table=rebalance_test_table_17 [id=9a4b9a0a2fe64ab2a64d6679185c2a33]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.016108 30256 tablet_service.cc:1467] Processing CreateTablet for tablet dd2fd47ec6684cb28d2f4678d2e90e75 (DEFAULT_TABLE table=rebalance_test_table_17 [id=9a4b9a0a2fe64ab2a64d6679185c2a33]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.016315 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a5869fb581b4939ad2207ab9200d674. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.016539 30125 tablet_service.cc:1467] Processing CreateTablet for tablet 4a5869fb581b4939ad2207ab9200d674 (DEFAULT_TABLE table=rebalance_test_table_17 [id=9a4b9a0a2fe64ab2a64d6679185c2a33]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.016711 30126 tablet_service.cc:1467] Processing CreateTablet for tablet dd2fd47ec6684cb28d2f4678d2e90e75 (DEFAULT_TABLE table=rebalance_test_table_17 [id=9a4b9a0a2fe64ab2a64d6679185c2a33]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.016784 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a5869fb581b4939ad2207ab9200d674. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.016893 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd2fd47ec6684cb28d2f4678d2e90e75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.018059 32126 tablet_bootstrap.cc:492] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:30.018790 32126 tablet_bootstrap.cc:654] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.018817 32124 tablet_bootstrap.cc:492] T dd2fd47ec6684cb28d2f4678d2e90e75 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:30.016539 30124 tablet_service.cc:1467] Processing CreateTablet for tablet 44949ddf1eac45cfa4fc8e1c532cf7bb (DEFAULT_TABLE table=rebalance_test_table_17 [id=9a4b9a0a2fe64ab2a64d6679185c2a33]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.019068 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44949ddf1eac45cfa4fc8e1c532cf7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.019814 32124 tablet_bootstrap.cc:654] T dd2fd47ec6684cb28d2f4678d2e90e75 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.020057 29995 tablet_service.cc:1467] Processing CreateTablet for tablet 4a5869fb581b4939ad2207ab9200d674 (DEFAULT_TABLE table=rebalance_test_table_17 [id=9a4b9a0a2fe64ab2a64d6679185c2a33]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.020336 29995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a5869fb581b4939ad2207ab9200d674. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.020681 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd2fd47ec6684cb28d2f4678d2e90e75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.020936 32124 tablet_bootstrap.cc:492] T dd2fd47ec6684cb28d2f4678d2e90e75 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:30.021009 32124 ts_tablet_manager.cc:1367] T dd2fd47ec6684cb28d2f4678d2e90e75 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:30.021481 32124 raft_consensus.cc:348] T dd2fd47ec6684cb28d2f4678d2e90e75 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:30.021577 32124 raft_consensus.cc:374] T dd2fd47ec6684cb28d2f4678d2e90e75 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.021613 32124 raft_consensus.cc:729] T dd2fd47ec6684cb28d2f4678d2e90e75 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.021754 32124 consensus_queue.cc:260] T dd2fd47ec6684cb28d2f4678d2e90e75 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:30.022007 32124 ts_tablet_manager.cc:1398] T dd2fd47ec6684cb28d2f4678d2e90e75 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:30.022176 32124 tablet_bootstrap.cc:492] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:30.022567 32017 tablet_bootstrap.cc:492] T 4a5869fb581b4939ad2207ab9200d674 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:30.023102 32124 tablet_bootstrap.cc:654] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.023617 32017 tablet_bootstrap.cc:654] T 4a5869fb581b4939ad2207ab9200d674 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.024439 30387 tablet_service.cc:1467] Processing CreateTablet for tablet dd2fd47ec6684cb28d2f4678d2e90e75 (DEFAULT_TABLE table=rebalance_test_table_17 [id=9a4b9a0a2fe64ab2a64d6679185c2a33]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.024705 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd2fd47ec6684cb28d2f4678d2e90e75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.024760 32017 tablet_bootstrap.cc:492] T 4a5869fb581b4939ad2207ab9200d674 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:30.024827 32017 ts_tablet_manager.cc:1367] T 4a5869fb581b4939ad2207ab9200d674 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:30.024857 30386 tablet_service.cc:1467] Processing CreateTablet for tablet 44949ddf1eac45cfa4fc8e1c532cf7bb (DEFAULT_TABLE table=rebalance_test_table_17 [id=9a4b9a0a2fe64ab2a64d6679185c2a33]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.025064 30386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44949ddf1eac45cfa4fc8e1c532cf7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.025290 32017 raft_consensus.cc:348] T 4a5869fb581b4939ad2207ab9200d674 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.025388 32017 raft_consensus.cc:374] T 4a5869fb581b4939ad2207ab9200d674 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.025416 32017 raft_consensus.cc:729] T 4a5869fb581b4939ad2207ab9200d674 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.025525 32017 consensus_queue.cc:260] T 4a5869fb581b4939ad2207ab9200d674 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.025808 32017 ts_tablet_manager.cc:1398] T 4a5869fb581b4939ad2207ab9200d674 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:30.026824 32123 tablet_bootstrap.cc:492] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:30.027663 32123 tablet_bootstrap.cc:654] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.027755 32126 tablet_bootstrap.cc:492] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:30.027835 32126 ts_tablet_manager.cc:1367] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:30.028347 32126 raft_consensus.cc:348] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.028451 32126 raft_consensus.cc:374] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.028496 32126 raft_consensus.cc:729] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.028613 32126 consensus_queue.cc:260] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.028867 32126 ts_tablet_manager.cc:1398] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:30.029068 32126 tablet_bootstrap.cc:492] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:30.029273 32123 tablet_bootstrap.cc:492] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:30.029341 32123 ts_tablet_manager.cc:1367] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:30.029775 32123 raft_consensus.cc:348] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:30.029886 32123 raft_consensus.cc:374] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.029924 32123 raft_consensus.cc:729] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.029953 32126 tablet_bootstrap.cc:654] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.030050 32123 consensus_queue.cc:260] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:30.030375 32123 ts_tablet_manager.cc:1398] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:30.030548 32123 tablet_bootstrap.cc:492] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:30.031174 32126 tablet_bootstrap.cc:492] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:30.031311 32126 ts_tablet_manager.cc:1367] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:30.031417 32123 tablet_bootstrap.cc:654] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.031973 32126 raft_consensus.cc:348] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.032131 32126 raft_consensus.cc:374] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.032212 32126 raft_consensus.cc:729] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.032521 32126 consensus_queue.cc:260] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.032989 32126 ts_tablet_manager.cc:1398] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:30.033160 32126 tablet_bootstrap.cc:492] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:30.034132 32126 tablet_bootstrap.cc:654] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.034144 32123 tablet_bootstrap.cc:492] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:30.034226 32123 ts_tablet_manager.cc:1367] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:30.034685 32123 raft_consensus.cc:348] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.034803 32123 raft_consensus.cc:374] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.034847 32123 raft_consensus.cc:729] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.034977 32123 consensus_queue.cc:260] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.035177 32126 tablet_bootstrap.cc:492] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:30.035255 32126 ts_tablet_manager.cc:1367] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:30.035270 32123 ts_tablet_manager.cc:1398] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:30.035630 32126 raft_consensus.cc:348] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:30.035724 32126 raft_consensus.cc:374] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.035753 32126 raft_consensus.cc:729] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.035852 32126 consensus_queue.cc:260] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:30.036186 32126 ts_tablet_manager.cc:1398] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:30.036943 32124 tablet_bootstrap.cc:492] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:30.037031 32124 ts_tablet_manager.cc:1367] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20240501 14:09:30.037372 32124 raft_consensus.cc:348] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.037498 32124 raft_consensus.cc:374] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.037540 32124 raft_consensus.cc:729] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.037644 32124 consensus_queue.cc:260] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.037946 32124 ts_tablet_manager.cc:1398] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:30.038179 32124 tablet_bootstrap.cc:492] T 4a5869fb581b4939ad2207ab9200d674 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:30.039073 32124 tablet_bootstrap.cc:654] T 4a5869fb581b4939ad2207ab9200d674 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.040903 32124 tablet_bootstrap.cc:492] T 4a5869fb581b4939ad2207ab9200d674 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:30.040985 32124 ts_tablet_manager.cc:1367] T 4a5869fb581b4939ad2207ab9200d674 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:09:30.041375 32124 raft_consensus.cc:348] T 4a5869fb581b4939ad2207ab9200d674 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.041469 32124 raft_consensus.cc:374] T 4a5869fb581b4939ad2207ab9200d674 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.041501 32124 raft_consensus.cc:729] T 4a5869fb581b4939ad2207ab9200d674 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.041620 32124 consensus_queue.cc:260] T 4a5869fb581b4939ad2207ab9200d674 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.041879 32124 ts_tablet_manager.cc:1398] T 4a5869fb581b4939ad2207ab9200d674 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:30.119124 32069 raft_consensus.cc:482] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:30.119241 32069 raft_consensus.cc:504] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:30.119601 32069 leader_election.cc:288] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:30.119928 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd2fd47ec6684cb28d2f4678d2e90e75" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:30.119933 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd2fd47ec6684cb28d2f4678d2e90e75" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:30.120065 30146 raft_consensus.cc:2454] T dd2fd47ec6684cb28d2f4678d2e90e75 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:30.120096 30264 raft_consensus.cc:2454] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:30.120369 30344 leader_election.cc:302] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:30.120517 32069 raft_consensus.cc:2781] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:30.120577 32069 raft_consensus.cc:482] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:30.120625 32069 raft_consensus.cc:3037] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.121498 32069 raft_consensus.cc:504] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:30.121819 32069 leader_election.cc:288] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:30.122210 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd2fd47ec6684cb28d2f4678d2e90e75" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:30.122287 30264 raft_consensus.cc:3037] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.122462 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd2fd47ec6684cb28d2f4678d2e90e75" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:30.122570 30146 raft_consensus.cc:3037] T dd2fd47ec6684cb28d2f4678d2e90e75 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.123118 30264 raft_consensus.cc:2454] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:30.123492 30146 raft_consensus.cc:2454] T dd2fd47ec6684cb28d2f4678d2e90e75 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:30.123536 30343 leader_election.cc:302] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:30.123775 32069 raft_consensus.cc:2781] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:30.123867 32069 raft_consensus.cc:686] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:30.124013 32069 consensus_queue.cc:237] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:30.126030 29870 catalog_manager.cc:5484] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:30.209527 31541 raft_consensus.cc:482] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:30.209646 31541 raft_consensus.cc:504] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.209985 31541 leader_election.cc:288] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:30.210297 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44949ddf1eac45cfa4fc8e1c532cf7bb" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:30.210325 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44949ddf1eac45cfa4fc8e1c532cf7bb" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:30.210440 30146 raft_consensus.cc:2454] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:30.210471 30406 raft_consensus.cc:2454] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:30.210692 30210 leader_election.cc:302] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:30.210875 31541 raft_consensus.cc:2781] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:30.210927 31541 raft_consensus.cc:482] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:30.210969 31541 raft_consensus.cc:3037] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.211760 31541 raft_consensus.cc:504] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.212069 31541 leader_election.cc:288] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:30.212348 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44949ddf1eac45cfa4fc8e1c532cf7bb" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:30.212375 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44949ddf1eac45cfa4fc8e1c532cf7bb" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:30.212438 30406 raft_consensus.cc:3037] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.212457 30146 raft_consensus.cc:3037] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.213121 30146 raft_consensus.cc:2454] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:30.213192 30406 raft_consensus.cc:2454] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:30.213377 30213 leader_election.cc:302] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:30.213519 31541 raft_consensus.cc:2781] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:30.213619 31541 raft_consensus.cc:686] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:30.213740 31541 consensus_queue.cc:237] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.215591 29870 catalog_manager.cc:5484] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:30.238173 31541 raft_consensus.cc:482] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:30.238276 31541 raft_consensus.cc:504] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.238557 31541 leader_election.cc:288] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:30.238844 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a5869fb581b4939ad2207ab9200d674" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:30.238865 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a5869fb581b4939ad2207ab9200d674" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:30.238971 30146 raft_consensus.cc:2454] T 4a5869fb581b4939ad2207ab9200d674 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:30.238982 30013 raft_consensus.cc:2454] T 4a5869fb581b4939ad2207ab9200d674 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:30.239226 30213 leader_election.cc:302] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:30.239430 31541 raft_consensus.cc:2781] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:30.239473 31541 raft_consensus.cc:482] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:30.239526 31541 raft_consensus.cc:3037] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.240339 31541 raft_consensus.cc:504] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.240626 31541 leader_election.cc:288] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:30.240900 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a5869fb581b4939ad2207ab9200d674" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:30.241019 30013 raft_consensus.cc:3037] T 4a5869fb581b4939ad2207ab9200d674 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.240986 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a5869fb581b4939ad2207ab9200d674" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:30.241097 30146 raft_consensus.cc:3037] T 4a5869fb581b4939ad2207ab9200d674 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.241761 30013 raft_consensus.cc:2454] T 4a5869fb581b4939ad2207ab9200d674 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:30.241833 30146 raft_consensus.cc:2454] T 4a5869fb581b4939ad2207ab9200d674 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:30.242035 30212 leader_election.cc:302] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:30.242178 31541 raft_consensus.cc:2781] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:30.242262 31541 raft_consensus.cc:686] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:30.242388 31541 consensus_queue.cc:237] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.244114 29870 catalog_manager.cc:5484] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
W20240501 14:09:30.367499 32187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:30.367722 32187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:30.370705 32187 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240501 14:09:30.476157 30277 raft_consensus.cc:1261] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:30.476262 30146 raft_consensus.cc:1261] T dd2fd47ec6684cb28d2f4678d2e90e75 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:30.476625 31874 consensus_queue.cc:1035] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:30.476915 31874 consensus_queue.cc:1035] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:30.480623 30406 raft_consensus.cc:1261] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:30.481057 31541 consensus_queue.cc:1035] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:30.483110 30145 raft_consensus.cc:1261] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:30.483503 31541 consensus_queue.cc:1035] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:30.489356 30145 raft_consensus.cc:1261] T 4a5869fb581b4939ad2207ab9200d674 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:30.489730 31541 consensus_queue.cc:1035] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:30.491483 30013 raft_consensus.cc:1261] T 4a5869fb581b4939ad2207ab9200d674 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:30.491871 31541 consensus_queue.cc:1035] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 134.162 ms
time per row: 0.0670812 ms
I20240501 14:09:30.546507 29870 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:30.551934 30386 tablet_service.cc:1467] Processing CreateTablet for tablet 727bad29bdb44df3b032de5d51c2b7fd (DEFAULT_TABLE table=rebalance_test_table_18 [id=e756cf3e07134acfb79003690692e338]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.552043 29995 tablet_service.cc:1467] Processing CreateTablet for tablet 727bad29bdb44df3b032de5d51c2b7fd (DEFAULT_TABLE table=rebalance_test_table_18 [id=e756cf3e07134acfb79003690692e338]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.552302 30386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 727bad29bdb44df3b032de5d51c2b7fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.552318 29995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 727bad29bdb44df3b032de5d51c2b7fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.552482 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 217878532a4642179f06645f3e073a23 (DEFAULT_TABLE table=rebalance_test_table_18 [id=e756cf3e07134acfb79003690692e338]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.552721 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217878532a4642179f06645f3e073a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.552809 30256 tablet_service.cc:1467] Processing CreateTablet for tablet 217878532a4642179f06645f3e073a23 (DEFAULT_TABLE table=rebalance_test_table_18 [id=e756cf3e07134acfb79003690692e338]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.553061 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217878532a4642179f06645f3e073a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.555269 30386 tablet_service.cc:1467] Processing CreateTablet for tablet 217878532a4642179f06645f3e073a23 (DEFAULT_TABLE table=rebalance_test_table_18 [id=e756cf3e07134acfb79003690692e338]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.555501 30386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217878532a4642179f06645f3e073a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.555527 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 205a0fda164d46a29a031070f1d0c369 (DEFAULT_TABLE table=rebalance_test_table_18 [id=e756cf3e07134acfb79003690692e338]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.555750 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205a0fda164d46a29a031070f1d0c369. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.556169 30125 tablet_service.cc:1467] Processing CreateTablet for tablet 727bad29bdb44df3b032de5d51c2b7fd (DEFAULT_TABLE table=rebalance_test_table_18 [id=e756cf3e07134acfb79003690692e338]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.556445 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 727bad29bdb44df3b032de5d51c2b7fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.556598 30124 tablet_service.cc:1467] Processing CreateTablet for tablet 205a0fda164d46a29a031070f1d0c369 (DEFAULT_TABLE table=rebalance_test_table_18 [id=e756cf3e07134acfb79003690692e338]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.556828 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205a0fda164d46a29a031070f1d0c369. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.557562 30387 tablet_service.cc:1467] Processing CreateTablet for tablet 205a0fda164d46a29a031070f1d0c369 (DEFAULT_TABLE table=rebalance_test_table_18 [id=e756cf3e07134acfb79003690692e338]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.557785 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205a0fda164d46a29a031070f1d0c369. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.558043 32237 tablet_bootstrap.cc:492] T 217878532a4642179f06645f3e073a23 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:30.559007 32237 tablet_bootstrap.cc:654] T 217878532a4642179f06645f3e073a23 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.559160 32238 tablet_bootstrap.cc:492] T 727bad29bdb44df3b032de5d51c2b7fd P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:30.559970 32238 tablet_bootstrap.cc:654] T 727bad29bdb44df3b032de5d51c2b7fd P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.560601 32236 tablet_bootstrap.cc:492] T 727bad29bdb44df3b032de5d51c2b7fd P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:30.560832 32235 tablet_bootstrap.cc:492] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:30.561614 32238 tablet_bootstrap.cc:492] T 727bad29bdb44df3b032de5d51c2b7fd P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:30.561722 32238 ts_tablet_manager.cc:1367] T 727bad29bdb44df3b032de5d51c2b7fd P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:30.562152 32238 raft_consensus.cc:348] T 727bad29bdb44df3b032de5d51c2b7fd P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.562268 32238 raft_consensus.cc:374] T 727bad29bdb44df3b032de5d51c2b7fd P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.562310 32238 raft_consensus.cc:729] T 727bad29bdb44df3b032de5d51c2b7fd P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.562434 32238 consensus_queue.cc:260] T 727bad29bdb44df3b032de5d51c2b7fd P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.562526 32236 tablet_bootstrap.cc:654] T 727bad29bdb44df3b032de5d51c2b7fd P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.562698 32238 ts_tablet_manager.cc:1398] T 727bad29bdb44df3b032de5d51c2b7fd P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:30.562839 32238 tablet_bootstrap.cc:492] T 205a0fda164d46a29a031070f1d0c369 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:30.563618 32238 tablet_bootstrap.cc:654] T 205a0fda164d46a29a031070f1d0c369 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.563932 32235 tablet_bootstrap.cc:654] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.564702 32238 tablet_bootstrap.cc:492] T 205a0fda164d46a29a031070f1d0c369 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:30.564831 32238 ts_tablet_manager.cc:1367] T 205a0fda164d46a29a031070f1d0c369 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:30.564936 32237 tablet_bootstrap.cc:492] T 217878532a4642179f06645f3e073a23 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:30.565021 32237 ts_tablet_manager.cc:1367] T 217878532a4642179f06645f3e073a23 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:30.565158 32235 tablet_bootstrap.cc:492] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:30.565245 32235 ts_tablet_manager.cc:1367] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:30.565275 32238 raft_consensus.cc:348] T 205a0fda164d46a29a031070f1d0c369 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:30.565368 32238 raft_consensus.cc:374] T 205a0fda164d46a29a031070f1d0c369 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.565440 32238 raft_consensus.cc:729] T 205a0fda164d46a29a031070f1d0c369 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.565428 32237 raft_consensus.cc:348] T 217878532a4642179f06645f3e073a23 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.565565 32237 raft_consensus.cc:374] T 217878532a4642179f06645f3e073a23 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.565606 32237 raft_consensus.cc:729] T 217878532a4642179f06645f3e073a23 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.565577 32238 consensus_queue.cc:260] T 205a0fda164d46a29a031070f1d0c369 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:30.565713 32237 consensus_queue.cc:260] T 217878532a4642179f06645f3e073a23 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.565997 32238 ts_tablet_manager.cc:1398] T 205a0fda164d46a29a031070f1d0c369 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:30.566175 32237 ts_tablet_manager.cc:1398] T 217878532a4642179f06645f3e073a23 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:30.565649 32235 raft_consensus.cc:348] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.566282 32235 raft_consensus.cc:374] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.566339 32235 raft_consensus.cc:729] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.566442 32235 consensus_queue.cc:260] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.566717 32235 ts_tablet_manager.cc:1398] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:30.566865 32235 tablet_bootstrap.cc:492] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:30.567539 32235 tablet_bootstrap.cc:654] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.569380 32236 tablet_bootstrap.cc:492] T 727bad29bdb44df3b032de5d51c2b7fd P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:30.569468 32236 ts_tablet_manager.cc:1367] T 727bad29bdb44df3b032de5d51c2b7fd P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:30.569578 32235 tablet_bootstrap.cc:492] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:30.569643 32235 ts_tablet_manager.cc:1367] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:30.569911 32235 raft_consensus.cc:348] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.569973 32235 raft_consensus.cc:374] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.569993 32235 raft_consensus.cc:729] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.569916 32236 raft_consensus.cc:348] T 727bad29bdb44df3b032de5d51c2b7fd P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.570025 32236 raft_consensus.cc:374] T 727bad29bdb44df3b032de5d51c2b7fd P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.570070 32236 raft_consensus.cc:729] T 727bad29bdb44df3b032de5d51c2b7fd P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.570081 32235 consensus_queue.cc:260] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.570187 32236 consensus_queue.cc:260] T 727bad29bdb44df3b032de5d51c2b7fd P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.570379 32235 ts_tablet_manager.cc:1398] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:30.570492 32236 ts_tablet_manager.cc:1398] T 727bad29bdb44df3b032de5d51c2b7fd P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:30.570664 32236 tablet_bootstrap.cc:492] T 217878532a4642179f06645f3e073a23 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:30.570897 32235 tablet_bootstrap.cc:492] T 205a0fda164d46a29a031070f1d0c369 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:30.571460 32236 tablet_bootstrap.cc:654] T 217878532a4642179f06645f3e073a23 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.571729 32235 tablet_bootstrap.cc:654] T 205a0fda164d46a29a031070f1d0c369 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.574756 32236 tablet_bootstrap.cc:492] T 217878532a4642179f06645f3e073a23 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:30.574749 32235 tablet_bootstrap.cc:492] T 205a0fda164d46a29a031070f1d0c369 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:30.574827 32235 ts_tablet_manager.cc:1367] T 205a0fda164d46a29a031070f1d0c369 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:30.574829 32236 ts_tablet_manager.cc:1367] T 217878532a4642179f06645f3e073a23 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:30.575124 32235 raft_consensus.cc:348] T 205a0fda164d46a29a031070f1d0c369 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:30.575194 32235 raft_consensus.cc:374] T 205a0fda164d46a29a031070f1d0c369 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.575223 32235 raft_consensus.cc:729] T 205a0fda164d46a29a031070f1d0c369 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.575243 32236 raft_consensus.cc:348] T 217878532a4642179f06645f3e073a23 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.575315 32236 raft_consensus.cc:374] T 217878532a4642179f06645f3e073a23 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.575346 32236 raft_consensus.cc:729] T 217878532a4642179f06645f3e073a23 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.575356 32235 consensus_queue.cc:260] T 205a0fda164d46a29a031070f1d0c369 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:30.575445 32236 consensus_queue.cc:260] T 217878532a4642179f06645f3e073a23 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.575589 32235 ts_tablet_manager.cc:1398] T 205a0fda164d46a29a031070f1d0c369 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:30.575701 32236 ts_tablet_manager.cc:1398] T 217878532a4642179f06645f3e073a23 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:30.575837 32236 tablet_bootstrap.cc:492] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:30.576653 32236 tablet_bootstrap.cc:654] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.577670 32236 tablet_bootstrap.cc:492] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:30.577746 32236 ts_tablet_manager.cc:1367] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:30.578132 32236 raft_consensus.cc:348] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:30.578230 32236 raft_consensus.cc:374] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.578284 32236 raft_consensus.cc:729] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.578413 32236 consensus_queue.cc:260] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:30.578663 32236 ts_tablet_manager.cc:1398] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:30.584645 31874 raft_consensus.cc:482] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:30.584720 31874 raft_consensus.cc:504] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:30.585022 31874 leader_election.cc:288] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:30.585295 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205a0fda164d46a29a031070f1d0c369" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:30.585309 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205a0fda164d46a29a031070f1d0c369" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:30.585400 30145 raft_consensus.cc:2454] T 205a0fda164d46a29a031070f1d0c369 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:30.585407 30013 raft_consensus.cc:2454] T 205a0fda164d46a29a031070f1d0c369 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:30.585634 30344 leader_election.cc:302] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:30.585820 31874 raft_consensus.cc:2781] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:30.585863 31874 raft_consensus.cc:482] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:30.585903 31874 raft_consensus.cc:3037] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.586714 31874 raft_consensus.cc:504] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:30.587060 31874 leader_election.cc:288] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:30.587335 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205a0fda164d46a29a031070f1d0c369" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:30.587432 30013 raft_consensus.cc:3037] T 205a0fda164d46a29a031070f1d0c369 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.587419 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205a0fda164d46a29a031070f1d0c369" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:30.587508 30145 raft_consensus.cc:3037] T 205a0fda164d46a29a031070f1d0c369 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.588115 30013 raft_consensus.cc:2454] T 205a0fda164d46a29a031070f1d0c369 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:30.588351 30145 raft_consensus.cc:2454] T 205a0fda164d46a29a031070f1d0c369 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:30.588403 30343 leader_election.cc:302] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:30.588589 31874 raft_consensus.cc:2781] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:30.588671 31874 raft_consensus.cc:686] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:30.588773 31874 consensus_queue.cc:237] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:30.590904 29871 catalog_manager.cc:5484] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:30.594832 32215 raft_consensus.cc:482] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:30.594916 32215 raft_consensus.cc:504] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.595216 32215 leader_election.cc:288] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:30.595480 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "727bad29bdb44df3b032de5d51c2b7fd" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:30.595579 30406 raft_consensus.cc:2454] T 727bad29bdb44df3b032de5d51c2b7fd P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:30.595784 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "727bad29bdb44df3b032de5d51c2b7fd" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:30.595835 29948 leader_election.cc:302] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:30.595888 30145 raft_consensus.cc:2454] T 727bad29bdb44df3b032de5d51c2b7fd P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:30.596002 32215 raft_consensus.cc:2781] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:30.596047 32215 raft_consensus.cc:482] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:30.596086 32215 raft_consensus.cc:3037] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.596959 32215 raft_consensus.cc:504] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.597275 32215 leader_election.cc:288] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:30.597545 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "727bad29bdb44df3b032de5d51c2b7fd" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:30.597640 30406 raft_consensus.cc:3037] T 727bad29bdb44df3b032de5d51c2b7fd P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.597713 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "727bad29bdb44df3b032de5d51c2b7fd" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:30.597798 30145 raft_consensus.cc:3037] T 727bad29bdb44df3b032de5d51c2b7fd P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.598533 30406 raft_consensus.cc:2454] T 727bad29bdb44df3b032de5d51c2b7fd P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:30.598660 30145 raft_consensus.cc:2454] T 727bad29bdb44df3b032de5d51c2b7fd P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:30.598848 29948 leader_election.cc:302] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:30.599112 32215 raft_consensus.cc:2781] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:30.599195 32215 raft_consensus.cc:686] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:30.599323 32215 consensus_queue.cc:237] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.600971 29871 catalog_manager.cc:5484] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:30.643376 31787 raft_consensus.cc:482] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:30.643500 31787 raft_consensus.cc:504] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.643873 31787 leader_election.cc:288] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:30.644284 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217878532a4642179f06645f3e073a23" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:30.644263 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217878532a4642179f06645f3e073a23" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:30.644405 30406 raft_consensus.cc:2454] T 217878532a4642179f06645f3e073a23 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:30.644412 30277 raft_consensus.cc:2454] T 217878532a4642179f06645f3e073a23 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:30.644739 29950 leader_election.cc:302] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:30.644953 31787 raft_consensus.cc:2781] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:30.645015 31787 raft_consensus.cc:482] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:30.645067 31787 raft_consensus.cc:3037] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.645833 31787 raft_consensus.cc:504] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.646118 31787 leader_election.cc:288] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:30.646400 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217878532a4642179f06645f3e073a23" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:30.646417 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217878532a4642179f06645f3e073a23" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:30.646507 30406 raft_consensus.cc:3037] T 217878532a4642179f06645f3e073a23 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.646508 30264 raft_consensus.cc:3037] T 217878532a4642179f06645f3e073a23 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.647218 30406 raft_consensus.cc:2454] T 217878532a4642179f06645f3e073a23 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:30.647358 30264 raft_consensus.cc:2454] T 217878532a4642179f06645f3e073a23 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:30.647452 29948 leader_election.cc:302] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:30.647609 31787 raft_consensus.cc:2781] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:30.647689 31787 raft_consensus.cc:686] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:30.647828 31787 consensus_queue.cc:237] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.649745 29871 catalog_manager.cc:5484] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
W20240501 14:09:30.771315 32251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:30.771546 32251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:30.774827 32251 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
I20240501 14:09:30.848526 30013 raft_consensus.cc:1261] T 205a0fda164d46a29a031070f1d0c369 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:30.849610 30145 raft_consensus.cc:1261] T 727bad29bdb44df3b032de5d51c2b7fd P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:30.849920 30146 raft_consensus.cc:1261] T 205a0fda164d46a29a031070f1d0c369 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:30.850106 30264 raft_consensus.cc:1261] T 217878532a4642179f06645f3e073a23 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:30.850551 30406 raft_consensus.cc:1261] T 727bad29bdb44df3b032de5d51c2b7fd P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:30.850597 32069 consensus_queue.cc:1035] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:30.850639 30407 raft_consensus.cc:1261] T 217878532a4642179f06645f3e073a23 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:30.850847 32069 consensus_queue.cc:1035] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:30.851256 31787 consensus_queue.cc:1035] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:30.851538 31787 consensus_queue.cc:1035] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:30.856201 32225 mvcc.cc:205] Tried to move back new op lower bound from 7022889250195853312 to 7022889249176150016. Current Snapshot: MvccSnapshot[applied={T|T < 7022889250195853312}]
I20240501 14:09:30.858748 32215 consensus_queue.cc:1035] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:30.859040 32215 consensus_queue.cc:1035] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 91.1201 ms
time per row: 0.0455601 ms
I20240501 14:09:30.909216 29871 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:30.914044 30256 tablet_service.cc:1467] Processing CreateTablet for tablet 19f203c753ef481dafc091ea2d60f56f (DEFAULT_TABLE table=rebalance_test_table_19 [id=00bf2d6ad5af4e6480096d5ceb4b1bb8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.914160 30124 tablet_service.cc:1467] Processing CreateTablet for tablet 19f203c753ef481dafc091ea2d60f56f (DEFAULT_TABLE table=rebalance_test_table_19 [id=00bf2d6ad5af4e6480096d5ceb4b1bb8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.914328 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f203c753ef481dafc091ea2d60f56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.914418 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f203c753ef481dafc091ea2d60f56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.914985 30125 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6a526c2ed84e1780ec392ef39d09b0 (DEFAULT_TABLE table=rebalance_test_table_19 [id=00bf2d6ad5af4e6480096d5ceb4b1bb8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.915186 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b6a526c2ed84e1780ec392ef39d09b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.916071 32238 tablet_bootstrap.cc:492] T 19f203c753ef481dafc091ea2d60f56f P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:30.916286 30387 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6a526c2ed84e1780ec392ef39d09b0 (DEFAULT_TABLE table=rebalance_test_table_19 [id=00bf2d6ad5af4e6480096d5ceb4b1bb8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.916460 32237 tablet_bootstrap.cc:492] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:30.916514 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b6a526c2ed84e1780ec392ef39d09b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.916905 32238 tablet_bootstrap.cc:654] T 19f203c753ef481dafc091ea2d60f56f P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.917294 32237 tablet_bootstrap.cc:654] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.918336 32237 tablet_bootstrap.cc:492] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:30.918433 32237 ts_tablet_manager.cc:1367] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:30.918846 30386 tablet_service.cc:1467] Processing CreateTablet for tablet 75b67ccb9a334b25ae71b553a2a84a02 (DEFAULT_TABLE table=rebalance_test_table_19 [id=00bf2d6ad5af4e6480096d5ceb4b1bb8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.918960 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 19f203c753ef481dafc091ea2d60f56f (DEFAULT_TABLE table=rebalance_test_table_19 [id=00bf2d6ad5af4e6480096d5ceb4b1bb8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.919049 30386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b67ccb9a334b25ae71b553a2a84a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.919209 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f203c753ef481dafc091ea2d60f56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.919126 32237 raft_consensus.cc:348] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:30.919246 32237 raft_consensus.cc:374] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.919277 32237 raft_consensus.cc:729] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.919389 32237 consensus_queue.cc:260] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:30.919674 32237 ts_tablet_manager.cc:1398] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:30.920673 32236 tablet_bootstrap.cc:492] T 3b6a526c2ed84e1780ec392ef39d09b0 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:30.921293 32235 tablet_bootstrap.cc:492] T 19f203c753ef481dafc091ea2d60f56f P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:30.921308 32236 tablet_bootstrap.cc:654] T 3b6a526c2ed84e1780ec392ef39d09b0 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.921433 29995 tablet_service.cc:1467] Processing CreateTablet for tablet 75b67ccb9a334b25ae71b553a2a84a02 (DEFAULT_TABLE table=rebalance_test_table_19 [id=00bf2d6ad5af4e6480096d5ceb4b1bb8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.921631 29995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b67ccb9a334b25ae71b553a2a84a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.922114 32235 tablet_bootstrap.cc:654] T 19f203c753ef481dafc091ea2d60f56f P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.922209 32236 tablet_bootstrap.cc:492] T 3b6a526c2ed84e1780ec392ef39d09b0 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:30.922264 32236 ts_tablet_manager.cc:1367] T 3b6a526c2ed84e1780ec392ef39d09b0 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:30.922616 32236 raft_consensus.cc:348] T 3b6a526c2ed84e1780ec392ef39d09b0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.922688 32236 raft_consensus.cc:374] T 3b6a526c2ed84e1780ec392ef39d09b0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.922709 32236 raft_consensus.cc:729] T 3b6a526c2ed84e1780ec392ef39d09b0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.922804 32236 consensus_queue.cc:260] T 3b6a526c2ed84e1780ec392ef39d09b0 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.923074 32236 ts_tablet_manager.cc:1398] T 3b6a526c2ed84e1780ec392ef39d09b0 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:30.923205 32236 tablet_bootstrap.cc:492] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:30.923964 32236 tablet_bootstrap.cc:654] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.924749 30255 tablet_service.cc:1467] Processing CreateTablet for tablet 75b67ccb9a334b25ae71b553a2a84a02 (DEFAULT_TABLE table=rebalance_test_table_19 [id=00bf2d6ad5af4e6480096d5ceb4b1bb8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.924837 30256 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6a526c2ed84e1780ec392ef39d09b0 (DEFAULT_TABLE table=rebalance_test_table_19 [id=00bf2d6ad5af4e6480096d5ceb4b1bb8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.925014 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b67ccb9a334b25ae71b553a2a84a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.925829 32236 tablet_bootstrap.cc:492] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:30.925918 32236 ts_tablet_manager.cc:1367] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:30.926318 32236 raft_consensus.cc:348] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.926407 32236 raft_consensus.cc:374] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.926437 32236 raft_consensus.cc:729] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.926577 32236 consensus_queue.cc:260] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.926903 32236 ts_tablet_manager.cc:1398] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:30.927528 32238 tablet_bootstrap.cc:492] T 19f203c753ef481dafc091ea2d60f56f P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:30.927606 32238 ts_tablet_manager.cc:1367] T 19f203c753ef481dafc091ea2d60f56f P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240501 14:09:30.928017 32238 raft_consensus.cc:348] T 19f203c753ef481dafc091ea2d60f56f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:30.928161 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b6a526c2ed84e1780ec392ef39d09b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.928159 32238 raft_consensus.cc:374] T 19f203c753ef481dafc091ea2d60f56f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.928208 32238 raft_consensus.cc:729] T 19f203c753ef481dafc091ea2d60f56f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.928359 32238 consensus_queue.cc:260] T 19f203c753ef481dafc091ea2d60f56f P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:30.928618 32238 ts_tablet_manager.cc:1398] T 19f203c753ef481dafc091ea2d60f56f P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:30.928761 32238 tablet_bootstrap.cc:492] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:30.928869 32235 tablet_bootstrap.cc:492] T 19f203c753ef481dafc091ea2d60f56f P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:30.928941 32235 ts_tablet_manager.cc:1367] T 19f203c753ef481dafc091ea2d60f56f P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:09:30.929347 32235 raft_consensus.cc:348] T 19f203c753ef481dafc091ea2d60f56f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:30.929457 32235 raft_consensus.cc:374] T 19f203c753ef481dafc091ea2d60f56f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.929505 32235 raft_consensus.cc:729] T 19f203c753ef481dafc091ea2d60f56f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.929531 32238 tablet_bootstrap.cc:654] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.929607 32235 consensus_queue.cc:260] T 19f203c753ef481dafc091ea2d60f56f P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:30.929884 32235 ts_tablet_manager.cc:1398] T 19f203c753ef481dafc091ea2d60f56f P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:30.930042 32235 tablet_bootstrap.cc:492] T 75b67ccb9a334b25ae71b553a2a84a02 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:30.930895 32235 tablet_bootstrap.cc:654] T 75b67ccb9a334b25ae71b553a2a84a02 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.931114 32238 tablet_bootstrap.cc:492] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:30.931187 32238 ts_tablet_manager.cc:1367] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:30.931630 32238 raft_consensus.cc:348] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.931727 32238 raft_consensus.cc:374] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.931757 32238 raft_consensus.cc:729] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.931871 32238 consensus_queue.cc:260] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.931926 32235 tablet_bootstrap.cc:492] T 75b67ccb9a334b25ae71b553a2a84a02 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:30.931998 32235 ts_tablet_manager.cc:1367] T 75b67ccb9a334b25ae71b553a2a84a02 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:30.932135 32238 ts_tablet_manager.cc:1398] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:30.932410 32235 raft_consensus.cc:348] T 75b67ccb9a334b25ae71b553a2a84a02 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.932524 32235 raft_consensus.cc:374] T 75b67ccb9a334b25ae71b553a2a84a02 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.932679 32235 raft_consensus.cc:729] T 75b67ccb9a334b25ae71b553a2a84a02 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.932816 32235 consensus_queue.cc:260] T 75b67ccb9a334b25ae71b553a2a84a02 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.932896 32237 tablet_bootstrap.cc:492] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:30.933138 32235 ts_tablet_manager.cc:1398] T 75b67ccb9a334b25ae71b553a2a84a02 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:30.933974 32237 tablet_bootstrap.cc:654] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.935735 32237 tablet_bootstrap.cc:492] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:30.935812 32237 ts_tablet_manager.cc:1367] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:30.936137 32237 raft_consensus.cc:348] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.936218 32237 raft_consensus.cc:374] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.936271 32237 raft_consensus.cc:729] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.936374 32237 consensus_queue.cc:260] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:30.936628 32237 ts_tablet_manager.cc:1398] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:30.938602 32237 tablet_bootstrap.cc:492] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:30.939334 32237 tablet_bootstrap.cc:654] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.940126 32237 tablet_bootstrap.cc:492] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:30.940184 32237 ts_tablet_manager.cc:1367] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:30.940601 32237 raft_consensus.cc:348] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.940680 32237 raft_consensus.cc:374] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.940739 32237 raft_consensus.cc:729] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.940842 32237 consensus_queue.cc:260] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:30.941149 32237 ts_tablet_manager.cc:1398] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:30.959231 32224 raft_consensus.cc:482] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:30.959324 32224 raft_consensus.cc:504] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:30.959667 32224 leader_election.cc:288] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:30.959944 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f203c753ef481dafc091ea2d60f56f" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:30.959960 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f203c753ef481dafc091ea2d60f56f" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:30.960053 30013 raft_consensus.cc:2454] T 19f203c753ef481dafc091ea2d60f56f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:30.960058 30146 raft_consensus.cc:2454] T 19f203c753ef481dafc091ea2d60f56f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:30.960327 30213 leader_election.cc:302] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:30.960469 32224 raft_consensus.cc:2781] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:30.960542 32224 raft_consensus.cc:482] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:30.960569 32224 raft_consensus.cc:3037] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.961335 32224 raft_consensus.cc:504] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:30.961603 32224 leader_election.cc:288] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:30.961872 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f203c753ef481dafc091ea2d60f56f" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:30.961884 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f203c753ef481dafc091ea2d60f56f" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:30.961979 30013 raft_consensus.cc:3037] T 19f203c753ef481dafc091ea2d60f56f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.961990 30146 raft_consensus.cc:3037] T 19f203c753ef481dafc091ea2d60f56f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.962862 30013 raft_consensus.cc:2454] T 19f203c753ef481dafc091ea2d60f56f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:30.962873 30146 raft_consensus.cc:2454] T 19f203c753ef481dafc091ea2d60f56f P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:30.963107 30212 leader_election.cc:302] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:30.963249 32224 raft_consensus.cc:2781] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:30.963320 32224 raft_consensus.cc:686] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:30.963438 32224 consensus_queue.cc:237] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:30.965328 29871 catalog_manager.cc:5484] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:31.064970 32116 raft_consensus.cc:482] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:31.065097 32116 raft_consensus.cc:504] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:31.065405 32116 leader_election.cc:288] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:31.065701 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6a526c2ed84e1780ec392ef39d09b0" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:31.065716 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6a526c2ed84e1780ec392ef39d09b0" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:31.065815 30407 raft_consensus.cc:2454] T 3b6a526c2ed84e1780ec392ef39d09b0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:31.065850 30146 raft_consensus.cc:2454] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:31.066052 30210 leader_election.cc:302] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:31.066213 32116 raft_consensus.cc:2781] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:31.066272 32116 raft_consensus.cc:482] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:31.066298 32116 raft_consensus.cc:3037] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:31.067165 32116 raft_consensus.cc:504] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:31.067453 32116 leader_election.cc:288] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:31.067720 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6a526c2ed84e1780ec392ef39d09b0" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:31.067762 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6a526c2ed84e1780ec392ef39d09b0" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:31.067826 30407 raft_consensus.cc:3037] T 3b6a526c2ed84e1780ec392ef39d09b0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:31.067871 30146 raft_consensus.cc:3037] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:31.068794 30407 raft_consensus.cc:2454] T 3b6a526c2ed84e1780ec392ef39d09b0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:31.068802 30146 raft_consensus.cc:2454] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:31.069059 30213 leader_election.cc:302] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:31.069216 32116 raft_consensus.cc:2781] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:31.069300 32116 raft_consensus.cc:686] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:31.069440 32116 consensus_queue.cc:237] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:31.071482 29871 catalog_manager.cc:5484] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:31.141726 32116 raft_consensus.cc:482] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:31.141865 32116 raft_consensus.cc:504] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:31.142299 32116 leader_election.cc:288] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:31.142524 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b67ccb9a334b25ae71b553a2a84a02" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:31.142617 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b67ccb9a334b25ae71b553a2a84a02" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:31.142655 30014 raft_consensus.cc:2454] T 75b67ccb9a334b25ae71b553a2a84a02 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:31.142750 30407 raft_consensus.cc:2454] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:31.142993 30212 leader_election.cc:302] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:31.143339 32116 raft_consensus.cc:2781] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:31.143395 32116 raft_consensus.cc:482] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:31.143433 32116 raft_consensus.cc:3037] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:31.144680 32116 raft_consensus.cc:504] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:31.144966 32116 leader_election.cc:288] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:31.145244 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b67ccb9a334b25ae71b553a2a84a02" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:31.145345 30014 raft_consensus.cc:3037] T 75b67ccb9a334b25ae71b553a2a84a02 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:31.145323 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b67ccb9a334b25ae71b553a2a84a02" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:31.145457 30407 raft_consensus.cc:3037] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:31.146095 30014 raft_consensus.cc:2454] T 75b67ccb9a334b25ae71b553a2a84a02 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:31.146291 30407 raft_consensus.cc:2454] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:31.146361 30212 leader_election.cc:302] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:31.146497 32116 raft_consensus.cc:2781] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:31.146581 32116 raft_consensus.cc:686] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:31.146742 32116 consensus_queue.cc:237] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:31.149014 29871 catalog_manager.cc:5484] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
W20240501 14:09:31.258159 32302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:31.258405 32302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:31.261540 32302 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240501 14:09:31.340170 32224 consensus_queue.cc:1035] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:31.343669 32224 consensus_queue.cc:1035] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:31.351884 30407 raft_consensus.cc:1261] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:31.352047 30014 raft_consensus.cc:1261] T 75b67ccb9a334b25ae71b553a2a84a02 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:31.352727 30146 raft_consensus.cc:1261] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:31.353083 32330 consensus_queue.cc:1035] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:31.354139 32224 consensus_queue.cc:1035] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:31.354347 30407 raft_consensus.cc:1261] T 3b6a526c2ed84e1780ec392ef39d09b0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:31.354401 32224 consensus_queue.cc:1035] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:31.355046 32224 consensus_queue.cc:1035] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:31.374287 32219 mvcc.cc:205] Tried to move back new op lower bound from 7022889252253691904 to 7022889251418411008. Current Snapshot: MvccSnapshot[applied={T|T < 7022889252253691904}]
INSERT report
rows total: 2000
time total: 111.959 ms
time per row: 0.0559795 ms
I20240501 14:09:31.420454 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.133:46265
--local_ip_for_outbound_sockets=127.22.252.133
--tserver_master_addrs=127.22.252.190:35675
--webserver_port=37943
--webserver_interface=127.22.252.133
--builtin_ntp_servers=127.22.252.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:09:31.530944 32341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:31.531189 32341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:31.531262 32341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:31.534624 32341 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:31.534713 32341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:31.534818 32341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.133
I20240501 14:09:31.538555 32341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.133:46265
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.252.133
--webserver_port=37943
--tserver_master_addrs=127.22.252.190:35675
--never_fsync=true
--heap_profile_path=/tmp/kudu.32341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.133
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:31.539364 32341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:31.540606 32341 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:31.549141 32349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:31.550117 32346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:31.551069 32347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:31.551949 32341 server_base.cc:1034] running on GCE node
I20240501 14:09:31.552346 32341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:31.552826 32341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:31.553977 32341 hybrid_clock.cc:648] HybridClock initialized: now 1714572571553965 us; error 23 us; skew 500 ppm
I20240501 14:09:31.555744 32341 webserver.cc:460] Webserver started at http://127.22.252.133:37943/ using document root <none> and password file <none>
I20240501 14:09:31.556277 32341 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:31.556351 32341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:31.559509 32341 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:31.561303 32355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:31.562181 32341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:31.562276 32341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a91029b90bf9415a89a2f70736759ad3"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-bkzb"
I20240501 14:09:31.562633 32341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:31.588030 32341 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:31.588672 32341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:31.588825 32341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:31.589461 32341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:31.590685 32341 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:31.590750 32341 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:31.590811 32341 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:31.590839 32341 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:31.601037 32341 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.133:46265
I20240501 14:09:31.601052 32468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.133:46265 every 8 connection(s)
I20240501 14:09:31.602061 32341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:09:31.606741 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 32341
I20240501 14:09:31.607213 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.134:33497
--local_ip_for_outbound_sockets=127.22.252.134
--tserver_master_addrs=127.22.252.190:35675
--webserver_port=38537
--webserver_interface=127.22.252.134
--builtin_ntp_servers=127.22.252.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:09:31.610476 32469 heartbeater.cc:347] Connected to a master server at 127.22.252.190:35675
I20240501 14:09:31.610719 32469 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:31.611265 32469 heartbeater.cc:511] Master 127.22.252.190:35675 requested a full tablet report, sending...
I20240501 14:09:31.612409 29868 ts_manager.cc:194] Re-registered known tserver with Master: a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:31.612936 29868 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.133:36855
W20240501 14:09:31.716269 32472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:31.716454 32472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:31.716501 32472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:31.719394 32472 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:31.719449 32472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:31.719547 32472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.134
I20240501 14:09:31.723156 32472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.134:33497
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.252.134
--webserver_port=38537
--tserver_master_addrs=127.22.252.190:35675
--never_fsync=true
--heap_profile_path=/tmp/kudu.32472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.134
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:31.723907 32472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:31.724936 32472 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:31.733461 32480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:31.734136 32478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:31.735229 32477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:31.736127 32472 server_base.cc:1034] running on GCE node
I20240501 14:09:31.736470 32472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:31.736971 32472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:31.738117 32472 hybrid_clock.cc:648] HybridClock initialized: now 1714572571738104 us; error 28 us; skew 500 ppm
I20240501 14:09:31.739858 32472 webserver.cc:460] Webserver started at http://127.22.252.134:38537/ using document root <none> and password file <none>
I20240501 14:09:31.740406 32472 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:31.740478 32472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:31.743728 32472 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:31.745565 32486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:31.746668 32472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:31.746752 32472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-bkzb"
I20240501 14:09:31.747201 32472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:31.776345 32472 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:31.776918 32472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:31.777061 32472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:31.777551 32472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:31.778378 32472 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:31.778421 32472 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:31.778474 32472 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:31.778502 32472 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:31.787251 32472 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.134:33497
I20240501 14:09:31.787283 32599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.134:33497 every 8 connection(s)
I20240501 14:09:31.788268 32472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:09:31.793148 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 32472
I20240501 14:09:31.793584 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.135:41915
--local_ip_for_outbound_sockets=127.22.252.135
--tserver_master_addrs=127.22.252.190:35675
--webserver_port=44261
--webserver_interface=127.22.252.135
--builtin_ntp_servers=127.22.252.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:09:31.801450 32600 heartbeater.cc:347] Connected to a master server at 127.22.252.190:35675
I20240501 14:09:31.801664 32600 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:31.802228 32600 heartbeater.cc:511] Master 127.22.252.190:35675 requested a full tablet report, sending...
I20240501 14:09:31.803929 29868 ts_manager.cc:194] Re-registered known tserver with Master: 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:31.804598 29868 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.134:45071
W20240501 14:09:31.903478 32603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:31.903650 32603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:31.903688 32603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:31.906555 32603 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:31.906610 32603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:31.906675 32603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.135
I20240501 14:09:31.910415 32603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.135:41915
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.252.135
--webserver_port=44261
--tserver_master_addrs=127.22.252.190:35675
--never_fsync=true
--heap_profile_path=/tmp/kudu.32603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.135
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:31.911224 32603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:31.912344 32603 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:31.920689 32609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:31.921411 32608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:31.922482 32611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:31.923349 32603 server_base.cc:1034] running on GCE node
I20240501 14:09:31.923658 32603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:31.924126 32603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:31.925279 32603 hybrid_clock.cc:648] HybridClock initialized: now 1714572571925265 us; error 26 us; skew 500 ppm
I20240501 14:09:31.927260 32603 webserver.cc:460] Webserver started at http://127.22.252.135:44261/ using document root <none> and password file <none>
I20240501 14:09:31.927768 32603 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:31.927840 32603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:31.931255 32603 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:31.932936 32617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:31.933923 32603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:31.934018 32603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "db935501e9104de5b15c8780681d6c38"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-bkzb"
I20240501 14:09:31.934512 32603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:31.956069 32603 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:31.956768 32603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:31.956928 32603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:31.957485 32603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:31.958479 32603 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:31.958535 32603 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:31.958596 32603 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:31.958618 32603 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:31.967353 32603 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.135:41915
I20240501 14:09:31.967487 32730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.135:41915 every 8 connection(s)
I20240501 14:09:31.968365 32603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:09:31.969221 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 32603
I20240501 14:09:31.969647 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.136:45937
--local_ip_for_outbound_sockets=127.22.252.136
--tserver_master_addrs=127.22.252.190:35675
--webserver_port=41581
--webserver_interface=127.22.252.136
--builtin_ntp_servers=127.22.252.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:09:31.979183 32731 heartbeater.cc:347] Connected to a master server at 127.22.252.190:35675
I20240501 14:09:31.979384 32731 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:31.979772 32731 heartbeater.cc:511] Master 127.22.252.190:35675 requested a full tablet report, sending...
I20240501 14:09:31.980901 29868 ts_manager.cc:194] Re-registered known tserver with Master: db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:31.981524 29868 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.135:56721
W20240501 14:09:32.082883 32734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:32.083097 32734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:32.083153 32734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:32.085985 32734 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:09:32.086038 32734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:32.086124 32734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.136
I20240501 14:09:32.089541 32734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.136:45937
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.252.136
--webserver_port=41581
--tserver_master_addrs=127.22.252.190:35675
--never_fsync=true
--heap_profile_path=/tmp/kudu.32734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.136
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:32.090301 32734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:32.091293 32734 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:32.100517 32741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:32.101478 32743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:32.102243 32740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:32.103024 32734 server_base.cc:1034] running on GCE node
I20240501 14:09:32.103329 32734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:32.103798 32734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:32.104935 32734 hybrid_clock.cc:648] HybridClock initialized: now 1714572572104924 us; error 24 us; skew 500 ppm
I20240501 14:09:32.106793 32734 webserver.cc:460] Webserver started at http://127.22.252.136:41581/ using document root <none> and password file <none>
I20240501 14:09:32.107381 32734 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:32.107460 32734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:32.110546 32734 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:32.112357 32749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:32.113288 32734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:32.113381 32734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-bkzb"
I20240501 14:09:32.113803 32734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:32.130908 32734 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:32.131424 32734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:32.131557 32734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:32.132078 32734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:32.132895 32734 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:32.132953 32734 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:32.133005 32734 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:32.133035 32734 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:32.141419 32734 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.136:45937
I20240501 14:09:32.141482 394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.136:45937 every 8 connection(s)
I20240501 14:09:32.142403 32734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:09:32.145365 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 32734
I20240501 14:09:32.148631 29868 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:32.151737 395 heartbeater.cc:347] Connected to a master server at 127.22.252.190:35675
I20240501 14:09:32.151978 395 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:32.152482 395 heartbeater.cc:511] Master 127.22.252.190:35675 requested a full tablet report, sending...
I20240501 14:09:32.153550 29866 ts_manager.cc:194] Re-registered known tserver with Master: 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:32.154067 29866 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.136:48683
I20240501 14:09:32.161422 29995 tablet_service.cc:1467] Processing CreateTablet for tablet d0b56d4f6ec3408cb49d8eaf81e1c035 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c7db511e6b3f409f8774eeb63a638332]), partition=
I20240501 14:09:32.161422 30125 tablet_service.cc:1467] Processing CreateTablet for tablet d0b56d4f6ec3408cb49d8eaf81e1c035 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c7db511e6b3f409f8774eeb63a638332]), partition=
I20240501 14:09:32.161674 29995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b56d4f6ec3408cb49d8eaf81e1c035. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.161674 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b56d4f6ec3408cb49d8eaf81e1c035. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.164330 409 tablet_bootstrap.cc:492] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:32.164764 410 tablet_bootstrap.cc:492] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:32.165248 409 tablet_bootstrap.cc:654] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.165908 410 tablet_bootstrap.cc:654] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.166256 409 tablet_bootstrap.cc:492] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:32.166339 409 ts_tablet_manager.cc:1367] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:32.166749 409 raft_consensus.cc:348] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:32.166867 409 raft_consensus.cc:374] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.166900 409 raft_consensus.cc:729] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.167022 409 consensus_queue.cc:260] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:32.167295 409 ts_tablet_manager.cc:1398] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:32.168028 410 tablet_bootstrap.cc:492] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:32.168107 410 ts_tablet_manager.cc:1367] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:32.168525 410 raft_consensus.cc:348] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:32.168634 410 raft_consensus.cc:374] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.168669 410 raft_consensus.cc:729] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.168781 410 consensus_queue.cc:260] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:32.169056 410 ts_tablet_manager.cc:1398] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:32.170357 329 tablet_service.cc:1467] Processing CreateTablet for tablet d0b56d4f6ec3408cb49d8eaf81e1c035 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c7db511e6b3f409f8774eeb63a638332]), partition=
I20240501 14:09:32.171078 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b56d4f6ec3408cb49d8eaf81e1c035. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.177053 413 tablet_bootstrap.cc:492] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:32.178987 413 tablet_bootstrap.cc:654] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.179590 413 log.cc:824] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:32.180871 413 tablet_bootstrap.cc:492] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:32.181016 413 ts_tablet_manager.cc:1367] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:09:32.184341 413 raft_consensus.cc:348] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:32.184506 413 raft_consensus.cc:374] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.184583 413 raft_consensus.cc:729] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.184970 413 consensus_queue.cc:260] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:32.185503 395 heartbeater.cc:503] Master 127.22.252.190:35675 was elected leader, sending a full tablet report...
I20240501 14:09:32.185669 413 ts_tablet_manager.cc:1398] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 14:09:32.269474 32173 raft_consensus.cc:482] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:32.269587 32173 raft_consensus.cc:504] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:32.270080 32173 leader_election.cc:288] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:32.270696 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b56d4f6ec3408cb49d8eaf81e1c035" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:32.270840 30014 raft_consensus.cc:2454] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:32.271126 30081 leader_election.cc:302] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:32.271412 32173 raft_consensus.cc:2781] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:32.271468 32173 raft_consensus.cc:482] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:32.271512 32173 raft_consensus.cc:3037] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:32.272608 32173 raft_consensus.cc:504] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:32.272926 32173 leader_election.cc:288] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:32.273551 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b56d4f6ec3408cb49d8eaf81e1c035" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:32.273662 30014 raft_consensus.cc:3037] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:32.274302 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b56d4f6ec3408cb49d8eaf81e1c035" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:32.274307 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b56d4f6ec3408cb49d8eaf81e1c035" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:32.274607 349 raft_consensus.cc:2454] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:32.274703 30014 raft_consensus.cc:2454] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:32.275094 30081 leader_election.cc:302] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters: 9ecd907cf5634ac98f3f54f3dbec6849
I20240501 14:09:32.275357 32173 raft_consensus.cc:2781] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:32.275445 32173 raft_consensus.cc:686] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:32.275564 32173 consensus_queue.cc:237] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:32.277828 29871 catalog_manager.cc:5484] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:32.392120 29870 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
W20240501 14:09:32.393560 396 tablet.cc:2369] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:32.398912 30014 raft_consensus.cc:1261] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:32.399516 32174 consensus_queue.cc:1035] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:32.399588 349 raft_consensus.cc:3037] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:32.400928 349 raft_consensus.cc:1261] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:32.402479 32173 consensus_queue.cc:1035] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:32.403420 29865 catalog_manager.cc:5576] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387) does not have the latest schema for tablet d0b56d4f6ec3408cb49d8eaf81e1c035 (table rebalancer_dynamic_table [id=c7db511e6b3f409f8774eeb63a638332]). Expected version 1 got 0
I20240501 14:09:32.405906 481 tablet.cc:1722] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:32.405612 479 tablet.cc:1722] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:32.407166 29866 catalog_manager.cc:5576] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937) does not have the latest schema for tablet d0b56d4f6ec3408cb49d8eaf81e1c035 (table rebalancer_dynamic_table [id=c7db511e6b3f409f8774eeb63a638332]). Expected version 1 got 0
I20240501 14:09:32.407483 30256 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:32.407896 29870 catalog_manager.cc:5576] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387) does not have the latest schema for tablet d0b56d4f6ec3408cb49d8eaf81e1c035 (table rebalancer_dynamic_table [id=c7db511e6b3f409f8774eeb63a638332]). Expected version 1 got 0
I20240501 14:09:32.409214 479 tablet.cc:1620] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3: MemRowSet was empty: no flush needed.
I20240501 14:09:32.414522 481 tablet.cc:1620] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e: MemRowSet was empty: no flush needed.
I20240501 14:09:32.415766 29870 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=c7db511e6b3f409f8774eeb63a638332] alter complete (version 1)
I20240501 14:09:32.419498 29866 catalog_manager.cc:5576] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937) does not have the latest schema for tablet d0b56d4f6ec3408cb49d8eaf81e1c035 (table rebalancer_dynamic_table [id=c7db511e6b3f409f8774eeb63a638332]). Expected version 1 got 0
I20240501 14:09:32.421263 478 mvcc.cc:205] Tried to move back new op lower bound from 7022889256541614080 to 7022889256041951232. Current Snapshot: MvccSnapshot[applied={T|T < 7022889256541614080}]
I20240501 14:09:32.425590 32534 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:32.427676 29995 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:32.428014 32401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:32.432418 30386 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:32.443920 483 tablet.cc:1722] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:32.444696 483 tablet.cc:1620] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:32.466105 329 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:32.466533 32665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:32.466843 30125 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:32.518487 29866 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:32.524392 479 tablet.cc:1722] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:32.524700 479 tablet.cc:1620] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3: MemRowSet was empty: no flush needed.
I20240501 14:09:32.525328 483 tablet.cc:1722] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:32.525465 481 tablet.cc:1722] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:32.525660 483 tablet.cc:1620] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:32.525772 481 tablet.cc:1620] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e: MemRowSet was empty: no flush needed.
I20240501 14:09:32.531121 29870 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=c7db511e6b3f409f8774eeb63a638332] alter complete (version 2)
I20240501 14:09:32.607654 32534 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:32.613551 30256 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:32.615296 32469 heartbeater.cc:503] Master 127.22.252.190:35675 was elected leader, sending a full tablet report...
I20240501 14:09:32.624506 32401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:32.629685 329 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:32.638231 29865 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:32.647176 29869 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:32.647257 29870 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 14:09:32.647414 29870 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 14:09:32.648458 30386 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:32.648650 479 tablet.cc:1722] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:32.650689 29870 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 27944e6f68e742b78ef5b350650aaee2
I20240501 14:09:32.651062 29870 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 985c116f66ec4a6cb10e3ec1d5f6366f
I20240501 14:09:32.651315 29870 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet e22ffa3b919b4b10995ba3a73157b221
I20240501 14:09:32.651564 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 27944e6f68e742b78ef5b350650aaee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:32.651791 29995 tablet_service.cc:1514] Processing DeleteTablet for tablet 27944e6f68e742b78ef5b350650aaee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:32.651791 29994 tablet_service.cc:1514] Processing DeleteTablet for tablet e22ffa3b919b4b10995ba3a73157b221 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:32.651877 30255 tablet_service.cc:1514] Processing DeleteTablet for tablet e22ffa3b919b4b10995ba3a73157b221 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:32.652170 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet 985c116f66ec4a6cb10e3ec1d5f6366f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:32.652330 545 tablet_replica.cc:330] T 27944e6f68e742b78ef5b350650aaee2 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:32.652362 30256 tablet_service.cc:1514] Processing DeleteTablet for tablet 27944e6f68e742b78ef5b350650aaee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:32.652485 544 tablet_replica.cc:330] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:32.652710 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 985c116f66ec4a6cb10e3ec1d5f6366f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:32.652884 545 raft_consensus.cc:2229] T 27944e6f68e742b78ef5b350650aaee2 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:32.653079 545 raft_consensus.cc:2258] T 27944e6f68e742b78ef5b350650aaee2 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.653097 30386 tablet_service.cc:1514] Processing DeleteTablet for tablet 985c116f66ec4a6cb10e3ec1d5f6366f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:32.653398 30387 tablet_service.cc:1514] Processing DeleteTablet for tablet e22ffa3b919b4b10995ba3a73157b221 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:32.653595 544 raft_consensus.cc:2229] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:32.653929 544 raft_consensus.cc:2258] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.654050 545 ts_tablet_manager.cc:1875] T 27944e6f68e742b78ef5b350650aaee2 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.654906 544 ts_tablet_manager.cc:1875] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.655157 483 tablet.cc:1722] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:32.655506 481 tablet.cc:1722] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:32.656446 545 ts_tablet_manager.cc:1888] T 27944e6f68e742b78ef5b350650aaee2 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:32.656541 545 log.cc:1196] T 27944e6f68e742b78ef5b350650aaee2 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/27944e6f68e742b78ef5b350650aaee2
I20240501 14:09:32.656873 545 ts_tablet_manager.cc:1909] T 27944e6f68e742b78ef5b350650aaee2 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:32.657729 544 ts_tablet_manager.cc:1888] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:32.657840 544 log.cc:1196] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/e22ffa3b919b4b10995ba3a73157b221
I20240501 14:09:32.658159 544 ts_tablet_manager.cc:1909] T e22ffa3b919b4b10995ba3a73157b221 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting consensus metadata
I20240501 14:09:32.658706 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 27944e6f68e742b78ef5b350650aaee2 (table rebalance_test_table_0 [id=4435db0f5a574d0285fbd28671f7b39d]) successfully deleted
I20240501 14:09:32.658953 545 tablet_replica.cc:330] T e22ffa3b919b4b10995ba3a73157b221 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:32.659091 545 raft_consensus.cc:2229] T e22ffa3b919b4b10995ba3a73157b221 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:32.659226 544 tablet_replica.cc:330] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:32.659227 545 raft_consensus.cc:2258] T e22ffa3b919b4b10995ba3a73157b221 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.659703 545 ts_tablet_manager.cc:1875] T e22ffa3b919b4b10995ba3a73157b221 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.659837 29870 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=c7db511e6b3f409f8774eeb63a638332] alter complete (version 3)
I20240501 14:09:32.660336 544 raft_consensus.cc:2229] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:32.660404 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet e22ffa3b919b4b10995ba3a73157b221 (table rebalance_test_table_0 [id=4435db0f5a574d0285fbd28671f7b39d]) successfully deleted
I20240501 14:09:32.660576 544 raft_consensus.cc:2258] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.661135 544 ts_tablet_manager.cc:1875] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.661792 545 ts_tablet_manager.cc:1888] T e22ffa3b919b4b10995ba3a73157b221 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:32.661863 545 log.cc:1196] T e22ffa3b919b4b10995ba3a73157b221 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/e22ffa3b919b4b10995ba3a73157b221
I20240501 14:09:32.662140 545 ts_tablet_manager.cc:1909] T e22ffa3b919b4b10995ba3a73157b221 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:32.662673 545 tablet_replica.cc:330] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:32.662707 30125 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:32.662812 545 raft_consensus.cc:2229] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:32.663048 545 raft_consensus.cc:2258] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.663530 545 ts_tablet_manager.cc:1875] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.663574 544 ts_tablet_manager.cc:1888] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:32.663682 544 log.cc:1196] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/27944e6f68e742b78ef5b350650aaee2
I20240501 14:09:32.663933 543 tablet_replica.cc:330] T 27944e6f68e742b78ef5b350650aaee2 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:32.663969 544 ts_tablet_manager.cc:1909] T 27944e6f68e742b78ef5b350650aaee2 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting consensus metadata
I20240501 14:09:32.664338 543 raft_consensus.cc:2229] T 27944e6f68e742b78ef5b350650aaee2 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:32.664600 543 raft_consensus.cc:2258] T 27944e6f68e742b78ef5b350650aaee2 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.665303 545 ts_tablet_manager.cc:1888] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:32.665375 545 log.cc:1196] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/985c116f66ec4a6cb10e3ec1d5f6366f
I20240501 14:09:32.665520 543 ts_tablet_manager.cc:1875] T 27944e6f68e742b78ef5b350650aaee2 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.665593 545 ts_tablet_manager.cc:1909] T 985c116f66ec4a6cb10e3ec1d5f6366f P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:32.665911 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet e22ffa3b919b4b10995ba3a73157b221 (table rebalance_test_table_0 [id=4435db0f5a574d0285fbd28671f7b39d]) successfully deleted
I20240501 14:09:32.666188 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet 27944e6f68e742b78ef5b350650aaee2 (table rebalance_test_table_0 [id=4435db0f5a574d0285fbd28671f7b39d]) successfully deleted
I20240501 14:09:32.666332 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 985c116f66ec4a6cb10e3ec1d5f6366f (table rebalance_test_table_0 [id=4435db0f5a574d0285fbd28671f7b39d]) successfully deleted
I20240501 14:09:32.666823 546 tablet_replica.cc:330] T 985c116f66ec4a6cb10e3ec1d5f6366f P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:32.667124 29993 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:32.667507 546 raft_consensus.cc:2229] T 985c116f66ec4a6cb10e3ec1d5f6366f P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:32.667742 546 raft_consensus.cc:2258] T 985c116f66ec4a6cb10e3ec1d5f6366f P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.668258 29993 tablet_service.cc:1467] Processing CreateTablet for tablet e122adbba7c749cd93c53fa07a8a8b84 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=25cba3c30d2b44599355b7317baedb1c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:32.668516 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e122adbba7c749cd93c53fa07a8a8b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.670647 30387 tablet_service.cc:1467] Processing CreateTablet for tablet e122adbba7c749cd93c53fa07a8a8b84 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=25cba3c30d2b44599355b7317baedb1c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:32.670884 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e122adbba7c749cd93c53fa07a8a8b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.670922 30256 tablet_service.cc:1467] Processing CreateTablet for tablet b7cf82d1a02a4e4a864570966fc073f0 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=25cba3c30d2b44599355b7317baedb1c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:32.671169 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7cf82d1a02a4e4a864570966fc073f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.675634 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 575e8cde863540b09a52fe590cbf2bfc (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=25cba3c30d2b44599355b7317baedb1c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:32.676602 32533 tablet_service.cc:1467] Processing CreateTablet for tablet e122adbba7c749cd93c53fa07a8a8b84 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=25cba3c30d2b44599355b7317baedb1c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:32.680485 32401 tablet_service.cc:1467] Processing CreateTablet for tablet 575e8cde863540b09a52fe590cbf2bfc (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=25cba3c30d2b44599355b7317baedb1c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:32.680730 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 575e8cde863540b09a52fe590cbf2bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.681200 32401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 575e8cde863540b09a52fe590cbf2bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.679920 32402 tablet_service.cc:1467] Processing CreateTablet for tablet b7cf82d1a02a4e4a864570966fc073f0 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=25cba3c30d2b44599355b7317baedb1c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:32.684263 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 575e8cde863540b09a52fe590cbf2bfc (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=25cba3c30d2b44599355b7317baedb1c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:32.684710 32402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7cf82d1a02a4e4a864570966fc073f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.685161 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 575e8cde863540b09a52fe590cbf2bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.688519 32664 tablet_service.cc:1467] Processing CreateTablet for tablet b7cf82d1a02a4e4a864570966fc073f0 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=25cba3c30d2b44599355b7317baedb1c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:32.688786 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7cf82d1a02a4e4a864570966fc073f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.689074 546 ts_tablet_manager.cc:1875] T 985c116f66ec4a6cb10e3ec1d5f6366f P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.691140 550 tablet_bootstrap.cc:492] T e122adbba7c749cd93c53fa07a8a8b84 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:32.692102 550 tablet_bootstrap.cc:654] T e122adbba7c749cd93c53fa07a8a8b84 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.692625 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e122adbba7c749cd93c53fa07a8a8b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.692606 543 ts_tablet_manager.cc:1888] T 27944e6f68e742b78ef5b350650aaee2 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:32.692701 543 log.cc:1196] T 27944e6f68e742b78ef5b350650aaee2 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/27944e6f68e742b78ef5b350650aaee2
I20240501 14:09:32.692716 552 tablet_bootstrap.cc:492] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:32.693018 543 ts_tablet_manager.cc:1909] T 27944e6f68e742b78ef5b350650aaee2 P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:32.693063 32663 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:32.693734 552 tablet_bootstrap.cc:654] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.694908 551 tablet_bootstrap.cc:492] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:32.695013 550 tablet_bootstrap.cc:492] T e122adbba7c749cd93c53fa07a8a8b84 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:32.695099 550 ts_tablet_manager.cc:1367] T e122adbba7c749cd93c53fa07a8a8b84 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:32.695521 550 raft_consensus.cc:348] T e122adbba7c749cd93c53fa07a8a8b84 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:32.695627 550 raft_consensus.cc:374] T e122adbba7c749cd93c53fa07a8a8b84 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.695657 550 raft_consensus.cc:729] T e122adbba7c749cd93c53fa07a8a8b84 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.695742 551 tablet_bootstrap.cc:654] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.697196 546 ts_tablet_manager.cc:1888] T 985c116f66ec4a6cb10e3ec1d5f6366f P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:32.697384 546 log.cc:1196] T 985c116f66ec4a6cb10e3ec1d5f6366f P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/985c116f66ec4a6cb10e3ec1d5f6366f
I20240501 14:09:32.699182 546 ts_tablet_manager.cc:1909] T 985c116f66ec4a6cb10e3ec1d5f6366f P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:32.699827 543 tablet_replica.cc:330] T 985c116f66ec4a6cb10e3ec1d5f6366f P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:32.699887 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 27944e6f68e742b78ef5b350650aaee2 (table rebalance_test_table_0 [id=4435db0f5a574d0285fbd28671f7b39d]) successfully deleted
I20240501 14:09:32.700395 543 raft_consensus.cc:2229] T 985c116f66ec4a6cb10e3ec1d5f6366f P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:32.700702 550 consensus_queue.cc:260] T e122adbba7c749cd93c53fa07a8a8b84 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:32.701135 543 raft_consensus.cc:2258] T 985c116f66ec4a6cb10e3ec1d5f6366f P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.701834 543 ts_tablet_manager.cc:1875] T 985c116f66ec4a6cb10e3ec1d5f6366f P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.702463 546 tablet_replica.cc:330] T e22ffa3b919b4b10995ba3a73157b221 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:32.702487 557 tablet_bootstrap.cc:492] T b7cf82d1a02a4e4a864570966fc073f0 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:32.702860 558 tablet_bootstrap.cc:492] T e122adbba7c749cd93c53fa07a8a8b84 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:32.702994 550 ts_tablet_manager.cc:1398] T e122adbba7c749cd93c53fa07a8a8b84 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:09:32.704514 543 ts_tablet_manager.cc:1888] T 985c116f66ec4a6cb10e3ec1d5f6366f P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:32.705015 546 raft_consensus.cc:2229] T e22ffa3b919b4b10995ba3a73157b221 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:32.705235 546 raft_consensus.cc:2258] T e22ffa3b919b4b10995ba3a73157b221 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.705798 553 tablet_bootstrap.cc:492] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:32.706028 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 985c116f66ec4a6cb10e3ec1d5f6366f (table rebalance_test_table_0 [id=4435db0f5a574d0285fbd28671f7b39d]) successfully deleted
I20240501 14:09:32.706537 546 ts_tablet_manager.cc:1875] T e22ffa3b919b4b10995ba3a73157b221 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.707046 552 tablet_bootstrap.cc:492] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:32.707139 552 ts_tablet_manager.cc:1367] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20240501 14:09:32.707548 552 raft_consensus.cc:348] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:32.707664 552 raft_consensus.cc:374] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.707698 552 raft_consensus.cc:729] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.707909 552 consensus_queue.cc:260] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:32.708909 552 ts_tablet_manager.cc:1398] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:32.710434 543 log.cc:1196] T 985c116f66ec4a6cb10e3ec1d5f6366f P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/985c116f66ec4a6cb10e3ec1d5f6366f
I20240501 14:09:32.711059 551 tablet_bootstrap.cc:492] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:32.712857 558 tablet_bootstrap.cc:654] T e122adbba7c749cd93c53fa07a8a8b84 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.713037 551 ts_tablet_manager.cc:1367] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 14:09:32.713377 553 tablet_bootstrap.cc:654] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.713557 551 raft_consensus.cc:348] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:32.713718 551 raft_consensus.cc:374] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.713789 551 raft_consensus.cc:729] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.713773 558 log.cc:824] T e122adbba7c749cd93c53fa07a8a8b84 P 5f8a0cfdfdca4148a75686dc88059cf3: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:32.713968 551 consensus_queue.cc:260] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:32.714107 553 log.cc:824] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:32.714365 551 ts_tablet_manager.cc:1398] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:32.714679 543 ts_tablet_manager.cc:1909] T 985c116f66ec4a6cb10e3ec1d5f6366f P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:32.714834 557 tablet_bootstrap.cc:654] T b7cf82d1a02a4e4a864570966fc073f0 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.715421 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 985c116f66ec4a6cb10e3ec1d5f6366f (table rebalance_test_table_0 [id=4435db0f5a574d0285fbd28671f7b39d]) successfully deleted
I20240501 14:09:32.715548 546 ts_tablet_manager.cc:1888] T e22ffa3b919b4b10995ba3a73157b221 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:32.715628 546 log.cc:1196] T e22ffa3b919b4b10995ba3a73157b221 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/e22ffa3b919b4b10995ba3a73157b221
I20240501 14:09:32.715904 546 ts_tablet_manager.cc:1909] T e22ffa3b919b4b10995ba3a73157b221 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:32.716493 557 log.cc:824] T b7cf82d1a02a4e4a864570966fc073f0 P db935501e9104de5b15c8780681d6c38: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:32.716681 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet e22ffa3b919b4b10995ba3a73157b221 (table rebalance_test_table_0 [id=4435db0f5a574d0285fbd28671f7b39d]) successfully deleted
I20240501 14:09:32.718355 553 tablet_bootstrap.cc:492] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:32.718529 553 ts_tablet_manager.cc:1367] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240501 14:09:32.722375 553 raft_consensus.cc:348] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:32.722573 553 raft_consensus.cc:374] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.722630 553 raft_consensus.cc:729] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.723045 553 consensus_queue.cc:260] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:32.723893 553 ts_tablet_manager.cc:1398] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 14:09:32.724298 553 tablet_bootstrap.cc:492] T b7cf82d1a02a4e4a864570966fc073f0 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:32.724973 557 tablet_bootstrap.cc:492] T b7cf82d1a02a4e4a864570966fc073f0 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:32.725139 557 ts_tablet_manager.cc:1367] T b7cf82d1a02a4e4a864570966fc073f0 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20240501 14:09:32.725231 553 tablet_bootstrap.cc:654] T b7cf82d1a02a4e4a864570966fc073f0 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.732547 557 raft_consensus.cc:348] T b7cf82d1a02a4e4a864570966fc073f0 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:32.732806 557 raft_consensus.cc:374] T b7cf82d1a02a4e4a864570966fc073f0 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.732863 557 raft_consensus.cc:729] T b7cf82d1a02a4e4a864570966fc073f0 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.733340 557 consensus_queue.cc:260] T b7cf82d1a02a4e4a864570966fc073f0 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:32.733985 32731 heartbeater.cc:503] Master 127.22.252.190:35675 was elected leader, sending a full tablet report...
I20240501 14:09:32.734233 557 ts_tablet_manager.cc:1398] T b7cf82d1a02a4e4a864570966fc073f0 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240501 14:09:32.734580 557 tablet_bootstrap.cc:492] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:32.735567 557 tablet_bootstrap.cc:654] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.736303 558 tablet_bootstrap.cc:492] T e122adbba7c749cd93c53fa07a8a8b84 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:32.736497 558 ts_tablet_manager.cc:1367] T e122adbba7c749cd93c53fa07a8a8b84 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20240501 14:09:32.736692 557 tablet_bootstrap.cc:492] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:32.736763 557 ts_tablet_manager.cc:1367] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:32.737421 557 raft_consensus.cc:348] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:32.737504 553 tablet_bootstrap.cc:492] T b7cf82d1a02a4e4a864570966fc073f0 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:32.737519 557 raft_consensus.cc:374] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.737546 557 raft_consensus.cc:729] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.737584 553 ts_tablet_manager.cc:1367] T b7cf82d1a02a4e4a864570966fc073f0 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240501 14:09:32.737689 557 consensus_queue.cc:260] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:32.737933 557 ts_tablet_manager.cc:1398] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:32.738106 553 raft_consensus.cc:348] T b7cf82d1a02a4e4a864570966fc073f0 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:32.738238 553 raft_consensus.cc:374] T b7cf82d1a02a4e4a864570966fc073f0 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.738291 553 raft_consensus.cc:729] T b7cf82d1a02a4e4a864570966fc073f0 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.738444 553 consensus_queue.cc:260] T b7cf82d1a02a4e4a864570966fc073f0 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:32.738688 553 ts_tablet_manager.cc:1398] T b7cf82d1a02a4e4a864570966fc073f0 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:32.741838 558 raft_consensus.cc:348] T e122adbba7c749cd93c53fa07a8a8b84 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:32.742033 558 raft_consensus.cc:374] T e122adbba7c749cd93c53fa07a8a8b84 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.742110 558 raft_consensus.cc:729] T e122adbba7c749cd93c53fa07a8a8b84 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.742645 558 consensus_queue.cc:260] T e122adbba7c749cd93c53fa07a8a8b84 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:32.743395 32600 heartbeater.cc:503] Master 127.22.252.190:35675 was elected leader, sending a full tablet report...
I20240501 14:09:32.743538 558 ts_tablet_manager.cc:1398] T e122adbba7c749cd93c53fa07a8a8b84 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240501 14:09:32.743762 558 tablet_bootstrap.cc:492] T 575e8cde863540b09a52fe590cbf2bfc P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:32.745199 558 tablet_bootstrap.cc:654] T 575e8cde863540b09a52fe590cbf2bfc P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.746567 558 tablet_bootstrap.cc:492] T 575e8cde863540b09a52fe590cbf2bfc P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:32.746718 558 ts_tablet_manager.cc:1367] T 575e8cde863540b09a52fe590cbf2bfc P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:32.747267 558 raft_consensus.cc:348] T 575e8cde863540b09a52fe590cbf2bfc P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:32.747426 558 raft_consensus.cc:374] T 575e8cde863540b09a52fe590cbf2bfc P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.747499 558 raft_consensus.cc:729] T 575e8cde863540b09a52fe590cbf2bfc P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.747674 558 consensus_queue.cc:260] T 575e8cde863540b09a52fe590cbf2bfc P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:32.747984 558 ts_tablet_manager.cc:1398] T 575e8cde863540b09a52fe590cbf2bfc P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:32.752492 32284 raft_consensus.cc:482] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:32.752601 32284 raft_consensus.cc:504] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:32.753042 32284 leader_election.cc:288] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:32.753409 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e122adbba7c749cd93c53fa07a8a8b84" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:32.753554 30014 raft_consensus.cc:2454] T e122adbba7c749cd93c53fa07a8a8b84 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:32.753852 30343 leader_election.cc:302] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:32.754062 32284 raft_consensus.cc:2781] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:32.754107 32284 raft_consensus.cc:482] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:32.754138 32284 raft_consensus.cc:3037] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:32.754876 32284 raft_consensus.cc:504] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:32.755254 32284 leader_election.cc:288] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:32.755555 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e122adbba7c749cd93c53fa07a8a8b84" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:32.755668 30014 raft_consensus.cc:3037] T e122adbba7c749cd93c53fa07a8a8b84 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:32.756502 30014 raft_consensus.cc:2454] T e122adbba7c749cd93c53fa07a8a8b84 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:32.756750 30343 leader_election.cc:302] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:32.756899 32284 raft_consensus.cc:2781] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:32.756992 32284 raft_consensus.cc:686] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:32.757112 32284 consensus_queue.cc:237] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:32.759019 29865 catalog_manager.cc:5484] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:32.764066 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e122adbba7c749cd93c53fa07a8a8b84" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:32.764066 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e122adbba7c749cd93c53fa07a8a8b84" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:32.764365 32553 raft_consensus.cc:3037] T e122adbba7c749cd93c53fa07a8a8b84 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:32.765462 32553 raft_consensus.cc:2454] T e122adbba7c749cd93c53fa07a8a8b84 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:32.772990 29865 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:32.773151 29865 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:32.775476 29865 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet d0b56d4f6ec3408cb49d8eaf81e1c035
I20240501 14:09:32.776273 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b56d4f6ec3408cb49d8eaf81e1c035 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:32.776470 545 tablet_replica.cc:330] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:32.776392 329 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b56d4f6ec3408cb49d8eaf81e1c035 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:32.776623 545 raft_consensus.cc:2229] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:32.776757 545 raft_consensus.cc:2258] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.776221 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b56d4f6ec3408cb49d8eaf81e1c035 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:32.776872 581 tablet_replica.cc:330] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:32.777032 543 tablet_replica.cc:330] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:32.777153 543 raft_consensus.cc:2229] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:32.777186 581 raft_consensus.cc:2229] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:32.777379 543 raft_consensus.cc:2258] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.777388 581 raft_consensus.cc:2258] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.777446 545 ts_tablet_manager.cc:1875] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.778108 543 ts_tablet_manager.cc:1875] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.778694 581 ts_tablet_manager.cc:1875] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.779764 545 ts_tablet_manager.cc:1888] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:32.779829 545 log.cc:1196] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/d0b56d4f6ec3408cb49d8eaf81e1c035
I20240501 14:09:32.780112 545 ts_tablet_manager.cc:1909] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:32.780115 543 ts_tablet_manager.cc:1888] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:32.780174 543 log.cc:1196] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/d0b56d4f6ec3408cb49d8eaf81e1c035
I20240501 14:09:32.780440 543 ts_tablet_manager.cc:1909] T d0b56d4f6ec3408cb49d8eaf81e1c035 P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:32.781253 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet d0b56d4f6ec3408cb49d8eaf81e1c035 (table rebalancer_dynamic_table_new_name [id=c7db511e6b3f409f8774eeb63a638332]) successfully deleted
I20240501 14:09:32.781417 581 ts_tablet_manager.cc:1888] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:32.781455 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet d0b56d4f6ec3408cb49d8eaf81e1c035 (table rebalancer_dynamic_table_new_name [id=c7db511e6b3f409f8774eeb63a638332]) successfully deleted
I20240501 14:09:32.781565 581 log.cc:1196] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/d0b56d4f6ec3408cb49d8eaf81e1c035
I20240501 14:09:32.781937 581 ts_tablet_manager.cc:1909] T d0b56d4f6ec3408cb49d8eaf81e1c035 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:09:32.783023 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet d0b56d4f6ec3408cb49d8eaf81e1c035 (table rebalancer_dynamic_table_new_name [id=c7db511e6b3f409f8774eeb63a638332]) successfully deleted
W20240501 14:09:32.789580 32601 tablet.cc:2369] T 575e8cde863540b09a52fe590cbf2bfc P 5f8a0cfdfdca4148a75686dc88059cf3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:09:32.853825 32470 tablet.cc:2369] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:32.876845 562 raft_consensus.cc:482] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:32.877055 562 raft_consensus.cc:504] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:32.877784 29866 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:32.878371 562 leader_election.cc:288] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:32.880995 564 raft_consensus.cc:482] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:32.881187 564 raft_consensus.cc:504] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:32.882427 564 leader_election.cc:288] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:32.884757 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "575e8cde863540b09a52fe590cbf2bfc" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:32.884997 32683 raft_consensus.cc:2454] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:32.885494 32356 leader_election.cc:302] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:32.885792 562 raft_consensus.cc:2781] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:32.885867 562 raft_consensus.cc:482] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:32.885921 562 raft_consensus.cc:3037] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:32.887097 562 raft_consensus.cc:504] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:32.887483 562 leader_election.cc:288] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:32.888062 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "575e8cde863540b09a52fe590cbf2bfc" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:32.888193 32683 raft_consensus.cc:3037] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:32.889626 32683 raft_consensus.cc:2454] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:32.890064 32356 leader_election.cc:302] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:32.890631 562 raft_consensus.cc:2781] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:32.890846 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 5862d4b94e314e92aeba097f84a46f6a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=771f8f7b55af4d5c8613dbcbcf7cf281]), partition=
I20240501 14:09:32.890861 562 raft_consensus.cc:686] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:32.891095 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5862d4b94e314e92aeba097f84a46f6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.891219 562 consensus_queue.cc:237] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:32.891404 329 tablet_service.cc:1467] Processing CreateTablet for tablet 5862d4b94e314e92aeba097f84a46f6a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=771f8f7b55af4d5c8613dbcbcf7cf281]), partition=
I20240501 14:09:32.891626 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5862d4b94e314e92aeba097f84a46f6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.893230 557 tablet_bootstrap.cc:492] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:32.894260 557 tablet_bootstrap.cc:654] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.895738 557 tablet_bootstrap.cc:492] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:32.895823 557 ts_tablet_manager.cc:1367] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:32.896108 32401 tablet_service.cc:1467] Processing CreateTablet for tablet 5862d4b94e314e92aeba097f84a46f6a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=771f8f7b55af4d5c8613dbcbcf7cf281]), partition=
I20240501 14:09:32.896298 557 raft_consensus.cc:348] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:32.896391 557 raft_consensus.cc:374] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.896423 557 raft_consensus.cc:729] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.896565 557 consensus_queue.cc:260] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:32.896576 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "575e8cde863540b09a52fe590cbf2bfc" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:32.896703 32553 raft_consensus.cc:2454] T 575e8cde863540b09a52fe590cbf2bfc P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:32.896838 557 ts_tablet_manager.cc:1398] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:32.896891 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "575e8cde863540b09a52fe590cbf2bfc" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:32.896983 32554 raft_consensus.cc:3037] T 575e8cde863540b09a52fe590cbf2bfc P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:32.897958 32554 raft_consensus.cc:2454] T 575e8cde863540b09a52fe590cbf2bfc P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:32.899199 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "575e8cde863540b09a52fe590cbf2bfc" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:32.899379 32554 raft_consensus.cc:2379] T 575e8cde863540b09a52fe590cbf2bfc P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db935501e9104de5b15c8780681d6c38 in current term 1: Already voted for candidate a91029b90bf9415a89a2f70736759ad3 in this term.
I20240501 14:09:32.902489 618 tablet_bootstrap.cc:492] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:32.903486 618 tablet_bootstrap.cc:654] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.905227 32401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5862d4b94e314e92aeba097f84a46f6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.906103 618 tablet_bootstrap.cc:492] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:32.906195 618 ts_tablet_manager.cc:1367] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:32.906661 618 raft_consensus.cc:348] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:32.906783 618 raft_consensus.cc:374] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.906809 618 raft_consensus.cc:729] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.906944 618 consensus_queue.cc:260] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:32.907227 618 ts_tablet_manager.cc:1398] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:32.907541 553 tablet_bootstrap.cc:492] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:32.908568 553 tablet_bootstrap.cc:654] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.910382 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "575e8cde863540b09a52fe590cbf2bfc" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:32.911020 32620 leader_election.cc:302] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db935501e9104de5b15c8780681d6c38; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, a91029b90bf9415a89a2f70736759ad3
I20240501 14:09:32.911603 564 raft_consensus.cc:2726] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:09:32.915393 553 tablet_bootstrap.cc:492] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:32.915493 553 ts_tablet_manager.cc:1367] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:32.915711 620 raft_consensus.cc:482] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:32.915808 620 raft_consensus.cc:504] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:32.915982 553 raft_consensus.cc:348] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:32.915987 29865 catalog_manager.cc:5484] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:32.916106 553 raft_consensus.cc:374] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.916146 553 raft_consensus.cc:729] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.916302 553 consensus_queue.cc:260] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:32.916929 620 leader_election.cc:288] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:32.918918 553 ts_tablet_manager.cc:1398] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:32.922961 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7cf82d1a02a4e4a864570966fc073f0" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:32.923138 32416 raft_consensus.cc:2454] T b7cf82d1a02a4e4a864570966fc073f0 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:32.923461 30212 leader_election.cc:302] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:32.924083 620 raft_consensus.cc:2781] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:32.924156 620 raft_consensus.cc:482] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:32.924214 620 raft_consensus.cc:3037] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:32.925283 620 raft_consensus.cc:504] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:32.925655 620 leader_election.cc:288] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:32.926203 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7cf82d1a02a4e4a864570966fc073f0" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:32.926312 32416 raft_consensus.cc:3037] T b7cf82d1a02a4e4a864570966fc073f0 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:32.927326 32416 raft_consensus.cc:2454] T b7cf82d1a02a4e4a864570966fc073f0 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:32.927628 30212 leader_election.cc:302] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:32.927841 620 raft_consensus.cc:2781] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:32.927949 620 raft_consensus.cc:686] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:32.928077 620 consensus_queue.cc:237] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:32.930199 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7cf82d1a02a4e4a864570966fc073f0" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:32.930353 32683 raft_consensus.cc:2454] T b7cf82d1a02a4e4a864570966fc073f0 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:32.930346 29867 catalog_manager.cc:5484] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:32.931017 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7cf82d1a02a4e4a864570966fc073f0" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:32.931133 32685 raft_consensus.cc:3037] T b7cf82d1a02a4e4a864570966fc073f0 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:32.934271 32685 raft_consensus.cc:2454] T b7cf82d1a02a4e4a864570966fc073f0 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:32.956177 29993 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:32.956807 32534 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:32.956876 30256 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:32.961710 329 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:32.965474 32401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:32.969734 30125 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:09:32.976409 32732 tablet.cc:2369] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:33.004803 32665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:33.007174 415 raft_consensus.cc:482] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:33.007300 415 raft_consensus.cc:504] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:33.008464 415 leader_election.cc:288] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:33.018203 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5862d4b94e314e92aeba097f84a46f6a" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:33.018383 32685 raft_consensus.cc:2454] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:33.018810 32750 leader_election.cc:302] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:33.019083 415 raft_consensus.cc:2781] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:33.019135 415 raft_consensus.cc:482] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:33.019163 415 raft_consensus.cc:3037] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.020377 415 raft_consensus.cc:504] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:33.020773 415 leader_election.cc:288] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:33.021128 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5862d4b94e314e92aeba097f84a46f6a" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:33.021304 32416 raft_consensus.cc:2454] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:33.021342 30387 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:33.021142 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5862d4b94e314e92aeba097f84a46f6a" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:33.021560 32415 raft_consensus.cc:3037] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.022159 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5862d4b94e314e92aeba097f84a46f6a" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:33.022310 32685 raft_consensus.cc:3037] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.022583 32415 raft_consensus.cc:2454] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:33.022852 32752 leader_election.cc:302] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:33.023113 415 raft_consensus.cc:2781] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:33.023273 32685 raft_consensus.cc:2454] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:33.023334 415 raft_consensus.cc:686] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:33.023591 415 consensus_queue.cc:237] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:33.027230 29867 catalog_manager.cc:5484] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:33.084096 30014 consensus_queue.cc:237] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.085783 30408 raft_consensus.cc:1261] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.086172 547 consensus_queue.cc:1035] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.086277 30277 raft_consensus.cc:1261] T af39c304693645269a6af59190e95cad P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.087286 547 consensus_queue.cc:1035] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.089298 658 raft_consensus.cc:2932] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:33.089727 30277 raft_consensus.cc:2932] T af39c304693645269a6af59190e95cad P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:33.091395 30408 raft_consensus.cc:2932] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
W20240501 14:09:33.092155 29950 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): Couldn't send request to peer 9ecd907cf5634ac98f3f54f3dbec6849. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af39c304693645269a6af59190e95cad. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.093030 29865 catalog_manager.cc:5484] T af39c304693645269a6af59190e95cad P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:33.115470 30146 consensus_queue.cc:237] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:33.117249 30277 raft_consensus.cc:1261] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.117388 30014 raft_consensus.cc:1261] T 796cef2485c94b76a5f3e1944b73dcf6 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.117833 32174 consensus_queue.cc:1035] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.118054 678 consensus_queue.cc:1035] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.120373 32174 raft_consensus.cc:2932] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:33.122268 30014 raft_consensus.cc:2932] T 796cef2485c94b76a5f3e1944b73dcf6 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:33.122784 30264 raft_consensus.cc:2932] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
W20240501 14:09:33.123340 30081 consensus_peers.cc:485] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 -> Peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): Couldn't send request to peer a91029b90bf9415a89a2f70736759ad3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 796cef2485c94b76a5f3e1944b73dcf6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.123708 29867 catalog_manager.cc:5484] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 reported cstate change: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:33.134461 29865 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:33.139292 32415 raft_consensus.cc:1261] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:33.139590 32685 raft_consensus.cc:1261] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:33.139962 415 consensus_queue.cc:1035] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:33.140156 640 consensus_queue.cc:1035] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:33.143752 483 tablet.cc:1722] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:33.144078 483 tablet.cc:1620] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:33.147001 29872 catalog_manager.cc:5576] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265) does not have the latest schema for tablet 5862d4b94e314e92aeba097f84a46f6a (table rebalancer_dynamic_table [id=771f8f7b55af4d5c8613dbcbcf7cf281]). Expected version 1 got 0
I20240501 14:09:33.149956 29865 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet 5862d4b94e314e92aeba097f84a46f6a (table rebalancer_dynamic_table [id=771f8f7b55af4d5c8613dbcbcf7cf281]). Expected version 1 got 0
I20240501 14:09:33.150290 29867 catalog_manager.cc:5576] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265) does not have the latest schema for tablet 5862d4b94e314e92aeba097f84a46f6a (table rebalancer_dynamic_table [id=771f8f7b55af4d5c8613dbcbcf7cf281]). Expected version 1 got 0
I20240501 14:09:33.151105 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=771f8f7b55af4d5c8613dbcbcf7cf281] alter complete (version 1)
I20240501 14:09:33.152732 701 tablet.cc:1722] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:33.153273 29864 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 14:09:33.153398 29864 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 14:09:33.153522 29865 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet 5862d4b94e314e92aeba097f84a46f6a (table rebalancer_dynamic_table [id=771f8f7b55af4d5c8613dbcbcf7cf281]). Expected version 1 got 0
I20240501 14:09:33.153685 701 tablet.cc:1620] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:33.155702 29864 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 048651987f8d4311b8b6bfb73b60ee90
I20240501 14:09:33.156021 29864 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet df5c7fba18cc4cccb8c61b0895c91340
I20240501 14:09:33.156158 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 048651987f8d4311b8b6bfb73b60ee90 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:33.156209 29864 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet d67171f19dcc4dc3bbe937b35b0f9456
I20240501 14:09:33.156366 543 tablet_replica.cc:330] T 048651987f8d4311b8b6bfb73b60ee90 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:33.156524 543 raft_consensus.cc:2229] T 048651987f8d4311b8b6bfb73b60ee90 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.156651 543 raft_consensus.cc:2258] T 048651987f8d4311b8b6bfb73b60ee90 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.157146 543 ts_tablet_manager.cc:1875] T 048651987f8d4311b8b6bfb73b60ee90 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.159322 543 ts_tablet_manager.cc:1888] T 048651987f8d4311b8b6bfb73b60ee90 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.159384 543 log.cc:1196] T 048651987f8d4311b8b6bfb73b60ee90 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/048651987f8d4311b8b6bfb73b60ee90
I20240501 14:09:33.160852 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet d67171f19dcc4dc3bbe937b35b0f9456 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:33.160985 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet 048651987f8d4311b8b6bfb73b60ee90 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:33.161188 30256 tablet_service.cc:1514] Processing DeleteTablet for tablet df5c7fba18cc4cccb8c61b0895c91340 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:33.161360 29995 tablet_service.cc:1514] Processing DeleteTablet for tablet df5c7fba18cc4cccb8c61b0895c91340 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:33.161413 544 tablet_replica.cc:330] T df5c7fba18cc4cccb8c61b0895c91340 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:33.161507 545 tablet_replica.cc:330] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:33.161545 544 raft_consensus.cc:2229] T df5c7fba18cc4cccb8c61b0895c91340 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.161638 545 raft_consensus.cc:2229] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:33.161680 544 raft_consensus.cc:2258] T df5c7fba18cc4cccb8c61b0895c91340 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.161859 545 raft_consensus.cc:2258] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.162215 544 ts_tablet_manager.cc:1875] T df5c7fba18cc4cccb8c61b0895c91340 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.162385 545 ts_tablet_manager.cc:1875] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.163949 545 ts_tablet_manager.cc:1888] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.163997 545 log.cc:1196] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/048651987f8d4311b8b6bfb73b60ee90
I20240501 14:09:33.164165 544 ts_tablet_manager.cc:1888] T df5c7fba18cc4cccb8c61b0895c91340 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.164199 545 ts_tablet_manager.cc:1909] T 048651987f8d4311b8b6bfb73b60ee90 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:33.164225 544 log.cc:1196] T df5c7fba18cc4cccb8c61b0895c91340 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/df5c7fba18cc4cccb8c61b0895c91340
I20240501 14:09:33.161000 29994 tablet_service.cc:1514] Processing DeleteTablet for tablet d67171f19dcc4dc3bbe937b35b0f9456 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:33.164495 544 ts_tablet_manager.cc:1909] T df5c7fba18cc4cccb8c61b0895c91340 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting consensus metadata
I20240501 14:09:33.164723 545 tablet_replica.cc:330] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:33.164840 545 raft_consensus.cc:2229] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:33.164935 30386 tablet_service.cc:1514] Processing DeleteTablet for tablet df5c7fba18cc4cccb8c61b0895c91340 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:33.164990 30387 tablet_service.cc:1514] Processing DeleteTablet for tablet 048651987f8d4311b8b6bfb73b60ee90 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:33.165097 546 tablet_replica.cc:330] T df5c7fba18cc4cccb8c61b0895c91340 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:33.165134 30388 tablet_service.cc:1514] Processing DeleteTablet for tablet d67171f19dcc4dc3bbe937b35b0f9456 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:33.165110 696 mvcc.cc:205] Tried to move back new op lower bound from 7022889259572322304 to 7022889259113328640. Current Snapshot: MvccSnapshot[applied={T|T < 7022889259572322304}]
I20240501 14:09:33.165201 546 raft_consensus.cc:2229] T df5c7fba18cc4cccb8c61b0895c91340 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.165288 546 raft_consensus.cc:2258] T df5c7fba18cc4cccb8c61b0895c91340 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.165400 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 048651987f8d4311b8b6bfb73b60ee90 (table rebalance_test_table_2 [id=2f015d5186014d24b820e1f454091285]) successfully deleted
I20240501 14:09:33.165747 546 ts_tablet_manager.cc:1875] T df5c7fba18cc4cccb8c61b0895c91340 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.166159 543 ts_tablet_manager.cc:1909] T 048651987f8d4311b8b6bfb73b60ee90 P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:33.166680 543 tablet_replica.cc:330] T d67171f19dcc4dc3bbe937b35b0f9456 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:33.166791 543 raft_consensus.cc:2229] T d67171f19dcc4dc3bbe937b35b0f9456 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.166818 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 048651987f8d4311b8b6bfb73b60ee90 (table rebalance_test_table_2 [id=2f015d5186014d24b820e1f454091285]) successfully deleted
I20240501 14:09:33.166889 543 raft_consensus.cc:2258] T d67171f19dcc4dc3bbe937b35b0f9456 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.167299 543 ts_tablet_manager.cc:1875] T d67171f19dcc4dc3bbe937b35b0f9456 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.167539 546 ts_tablet_manager.cc:1888] T df5c7fba18cc4cccb8c61b0895c91340 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.167587 546 log.cc:1196] T df5c7fba18cc4cccb8c61b0895c91340 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/df5c7fba18cc4cccb8c61b0895c91340
I20240501 14:09:33.167773 546 ts_tablet_manager.cc:1909] T df5c7fba18cc4cccb8c61b0895c91340 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:33.168552 546 tablet_replica.cc:330] T 048651987f8d4311b8b6bfb73b60ee90 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:33.168625 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet df5c7fba18cc4cccb8c61b0895c91340 (table rebalance_test_table_2 [id=2f015d5186014d24b820e1f454091285]) successfully deleted
I20240501 14:09:33.168678 546 raft_consensus.cc:2229] T 048651987f8d4311b8b6bfb73b60ee90 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.168787 546 raft_consensus.cc:2258] T 048651987f8d4311b8b6bfb73b60ee90 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.165019 545 raft_consensus.cc:2258] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.169309 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet df5c7fba18cc4cccb8c61b0895c91340 (table rebalance_test_table_2 [id=2f015d5186014d24b820e1f454091285]) successfully deleted
I20240501 14:09:33.169657 545 ts_tablet_manager.cc:1875] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.169870 543 ts_tablet_manager.cc:1888] T d67171f19dcc4dc3bbe937b35b0f9456 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.169929 543 log.cc:1196] T d67171f19dcc4dc3bbe937b35b0f9456 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/d67171f19dcc4dc3bbe937b35b0f9456
I20240501 14:09:33.170171 543 ts_tablet_manager.cc:1909] T d67171f19dcc4dc3bbe937b35b0f9456 P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:33.170820 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet d67171f19dcc4dc3bbe937b35b0f9456 (table rebalance_test_table_2 [id=2f015d5186014d24b820e1f454091285]) successfully deleted
I20240501 14:09:33.170818 702 tablet.cc:1722] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:33.171329 545 ts_tablet_manager.cc:1888] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.171376 545 log.cc:1196] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/df5c7fba18cc4cccb8c61b0895c91340
I20240501 14:09:33.171558 545 ts_tablet_manager.cc:1909] T df5c7fba18cc4cccb8c61b0895c91340 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:33.171718 546 ts_tablet_manager.cc:1875] T 048651987f8d4311b8b6bfb73b60ee90 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.171859 702 tablet.cc:1620] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:33.173235 546 ts_tablet_manager.cc:1888] T 048651987f8d4311b8b6bfb73b60ee90 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.173298 546 log.cc:1196] T 048651987f8d4311b8b6bfb73b60ee90 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/048651987f8d4311b8b6bfb73b60ee90
I20240501 14:09:33.173602 546 ts_tablet_manager.cc:1909] T 048651987f8d4311b8b6bfb73b60ee90 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:33.174283 546 tablet_replica.cc:330] T d67171f19dcc4dc3bbe937b35b0f9456 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:33.174436 546 raft_consensus.cc:2229] T d67171f19dcc4dc3bbe937b35b0f9456 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.174556 546 raft_consensus.cc:2258] T d67171f19dcc4dc3bbe937b35b0f9456 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.174541 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 048651987f8d4311b8b6bfb73b60ee90 (table rebalance_test_table_2 [id=2f015d5186014d24b820e1f454091285]) successfully deleted
I20240501 14:09:33.175005 546 ts_tablet_manager.cc:1875] T d67171f19dcc4dc3bbe937b35b0f9456 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.176544 545 tablet_replica.cc:330] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:33.176651 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet df5c7fba18cc4cccb8c61b0895c91340 (table rebalance_test_table_2 [id=2f015d5186014d24b820e1f454091285]) successfully deleted
I20240501 14:09:33.176811 545 raft_consensus.cc:2229] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:33.177460 30145 consensus_queue.cc:237] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:33.177570 545 raft_consensus.cc:2258] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.178165 545 ts_tablet_manager.cc:1875] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.179427 30014 raft_consensus.cc:1261] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.179525 30408 raft_consensus.cc:1261] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.179822 500 consensus_queue.cc:1035] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.180145 500 consensus_queue.cc:1035] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.181382 546 ts_tablet_manager.cc:1888] T d67171f19dcc4dc3bbe937b35b0f9456 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.181491 546 log.cc:1196] T d67171f19dcc4dc3bbe937b35b0f9456 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/d67171f19dcc4dc3bbe937b35b0f9456
I20240501 14:09:33.181828 546 ts_tablet_manager.cc:1909] T d67171f19dcc4dc3bbe937b35b0f9456 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:33.182344 545 ts_tablet_manager.cc:1888] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.182425 545 log.cc:1196] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/d67171f19dcc4dc3bbe937b35b0f9456
I20240501 14:09:33.182814 545 ts_tablet_manager.cc:1909] T d67171f19dcc4dc3bbe937b35b0f9456 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:33.182816 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet d67171f19dcc4dc3bbe937b35b0f9456 (table rebalance_test_table_2 [id=2f015d5186014d24b820e1f454091285]) successfully deleted
W20240501 14:09:33.183270 30082 consensus_peers.cc:485] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d5e1afe21654080b5aa3eb877e8f1d9. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.183421 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet d67171f19dcc4dc3bbe937b35b0f9456 (table rebalance_test_table_2 [id=2f015d5186014d24b820e1f454091285]) successfully deleted
I20240501 14:09:33.184881 678 raft_consensus.cc:2932] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:33.185386 30014 raft_consensus.cc:2932] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:33.190392 29864 catalog_manager.cc:5484] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:33.198853 30408 raft_consensus.cc:2932] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:33.201349 32735 consensus_queue.cc:1035] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:33.206137 32735 consensus_queue.cc:1035] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:33.233614 30408 consensus_queue.cc:237] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:33.235203 30145 raft_consensus.cc:1261] T dd2fd47ec6684cb28d2f4678d2e90e75 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.235663 30277 raft_consensus.cc:1261] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.235654 728 consensus_queue.cc:1035] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.236276 728 consensus_queue.cc:1035] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.239603 728 raft_consensus.cc:2932] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
W20240501 14:09:33.240129 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd2fd47ec6684cb28d2f4678d2e90e75. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.240885 30145 raft_consensus.cc:2932] T dd2fd47ec6684cb28d2f4678d2e90e75 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:33.241870 30277 raft_consensus.cc:2932] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:33.244388 29865 catalog_manager.cc:5484] T dd2fd47ec6684cb28d2f4678d2e90e75 P d35c16dcb506426993811bb2519d7db3 reported cstate change: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:33.252955 29867 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:33.258999 483 tablet.cc:1722] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:33.259353 483 tablet.cc:1620] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:33.259765 702 tablet.cc:1722] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:33.260116 702 tablet.cc:1620] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:33.262719 701 tablet.cc:1722] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:33.263090 701 tablet.cc:1620] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:33.269398 29867 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=771f8f7b55af4d5c8613dbcbcf7cf281] alter complete (version 2)
I20240501 14:09:33.275034 30407 consensus_queue.cc:237] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.277565 30277 raft_consensus.cc:1261] T 940c8f54339f4a32be2942682a41c685 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.277937 32284 consensus_queue.cc:1035] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.279016 30145 raft_consensus.cc:1261] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.279459 32284 consensus_queue.cc:1035] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:33.281121 30343 consensus_peers.cc:485] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): Couldn't send request to peer 9ecd907cf5634ac98f3f54f3dbec6849. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 940c8f54339f4a32be2942682a41c685. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.281766 32735 raft_consensus.cc:2932] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:33.282274 30145 raft_consensus.cc:2932] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:33.283844 30277 raft_consensus.cc:2932] T 940c8f54339f4a32be2942682a41c685 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:33.284853 29865 catalog_manager.cc:5484] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:33.291307 562 consensus_queue.cc:1035] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:33.298774 617 consensus_queue.cc:1035] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:33.312742 30407 consensus_queue.cc:237] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:33.314072 30014 raft_consensus.cc:1261] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.314422 30277 raft_consensus.cc:1261] T 4f5314fbd59c4443974a9c304ba9b1a7 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.314807 728 consensus_queue.cc:1035] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.315135 728 consensus_queue.cc:1035] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:33.318341 30343 consensus_peers.cc:485] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): Couldn't send request to peer a91029b90bf9415a89a2f70736759ad3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f5314fbd59c4443974a9c304ba9b1a7. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.321179 728 raft_consensus.cc:2932] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:33.321687 30014 raft_consensus.cc:2932] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:33.322897 30277 raft_consensus.cc:2932] T 4f5314fbd59c4443974a9c304ba9b1a7 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:33.326548 29867 catalog_manager.cc:5484] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:33.347261 30014 consensus_queue.cc:237] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:33.349375 30407 raft_consensus.cc:1261] T 5da2beba68ba414281025fcee91ca0d0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.349767 30146 raft_consensus.cc:1261] T 5da2beba68ba414281025fcee91ca0d0 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.350178 547 consensus_queue.cc:1035] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.350382 32287 consensus_queue.cc:1035] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.353260 547 raft_consensus.cc:2932] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:33.354828 30407 raft_consensus.cc:2932] T 5da2beba68ba414281025fcee91ca0d0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
W20240501 14:09:33.355245 29951 consensus_peers.cc:485] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5da2beba68ba414281025fcee91ca0d0. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.355616 30146 raft_consensus.cc:2932] T 5da2beba68ba414281025fcee91ca0d0 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:33.358297 29865 catalog_manager.cc:5484] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:33.372813 29867 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:33.380179 483 tablet.cc:1722] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:33.381057 702 tablet.cc:1722] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:33.382714 701 tablet.cc:1722] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:33.386994 29867 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=771f8f7b55af4d5c8613dbcbcf7cf281] alter complete (version 3)
I20240501 14:09:33.392833 30407 consensus_queue.cc:237] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
W20240501 14:09:33.394377 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a04e958f3d049139ff74202611af725. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.395735 30145 raft_consensus.cc:1261] T 3a04e958f3d049139ff74202611af725 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.396025 30277 raft_consensus.cc:1261] T 3a04e958f3d049139ff74202611af725 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.396333 32735 consensus_queue.cc:1035] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.396522 731 consensus_queue.cc:1035] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.398865 32735 raft_consensus.cc:2932] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:33.399924 30277 raft_consensus.cc:2932] T 3a04e958f3d049139ff74202611af725 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:33.400796 30145 raft_consensus.cc:2932] T 3a04e958f3d049139ff74202611af725 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:33.402989 29864 catalog_manager.cc:5484] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:33.405689 32330 consensus_queue.cc:1035] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:33.416353 620 consensus_queue.cc:1035] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:33.436748 30145 consensus_queue.cc:237] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.438295 30014 raft_consensus.cc:1261] T 8d87b2a4f6764e20afbfb43681f58467 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.438750 706 consensus_queue.cc:1035] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:33.440551 30081 consensus_peers.cc:485] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 -> Peer 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): Couldn't send request to peer 9ecd907cf5634ac98f3f54f3dbec6849. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d87b2a4f6764e20afbfb43681f58467. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.441058 30277 raft_consensus.cc:1261] T 8d87b2a4f6764e20afbfb43681f58467 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.441553 32174 consensus_queue.cc:1035] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.445264 706 raft_consensus.cc:2932] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:33.446439 30014 raft_consensus.cc:2932] T 8d87b2a4f6764e20afbfb43681f58467 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:33.446962 29864 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:33.447131 30277 raft_consensus.cc:2932] T 8d87b2a4f6764e20afbfb43681f58467 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:33.450745 29865 catalog_manager.cc:5484] T 8d87b2a4f6764e20afbfb43681f58467 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:33.460742 329 tablet_service.cc:1467] Processing CreateTablet for tablet 21efaac3a3d643679c5abaaddb9e5a3a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e53010e4736c42a899edac137f9182cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:33.461164 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21efaac3a3d643679c5abaaddb9e5a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.461318 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 21efaac3a3d643679c5abaaddb9e5a3a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e53010e4736c42a899edac137f9182cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:33.461318 32533 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5c502325fb475f8f050ccad34feb2a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e53010e4736c42a899edac137f9182cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:33.461619 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21efaac3a3d643679c5abaaddb9e5a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.461786 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5c502325fb475f8f050ccad34feb2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.464164 837 tablet_bootstrap.cc:492] T 21efaac3a3d643679c5abaaddb9e5a3a P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:33.465258 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 21efaac3a3d643679c5abaaddb9e5a3a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e53010e4736c42a899edac137f9182cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:33.465516 837 tablet_bootstrap.cc:654] T 21efaac3a3d643679c5abaaddb9e5a3a P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.465602 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21efaac3a3d643679c5abaaddb9e5a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.466537 837 tablet_bootstrap.cc:492] T 21efaac3a3d643679c5abaaddb9e5a3a P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:33.466614 837 ts_tablet_manager.cc:1367] T 21efaac3a3d643679c5abaaddb9e5a3a P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:33.467108 837 raft_consensus.cc:348] T 21efaac3a3d643679c5abaaddb9e5a3a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:33.467206 837 raft_consensus.cc:374] T 21efaac3a3d643679c5abaaddb9e5a3a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.467233 837 raft_consensus.cc:729] T 21efaac3a3d643679c5abaaddb9e5a3a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.467342 837 consensus_queue.cc:260] T 21efaac3a3d643679c5abaaddb9e5a3a P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:33.467567 837 ts_tablet_manager.cc:1398] T 21efaac3a3d643679c5abaaddb9e5a3a P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:33.468480 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5c502325fb475f8f050ccad34feb2a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e53010e4736c42a899edac137f9182cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:33.468752 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5c502325fb475f8f050ccad34feb2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.464565 836 tablet_bootstrap.cc:492] T 2c5c502325fb475f8f050ccad34feb2a P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:33.470549 836 tablet_bootstrap.cc:654] T 2c5c502325fb475f8f050ccad34feb2a P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.471479 836 tablet_bootstrap.cc:492] T 2c5c502325fb475f8f050ccad34feb2a P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:33.471545 836 ts_tablet_manager.cc:1367] T 2c5c502325fb475f8f050ccad34feb2a P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:33.471906 836 raft_consensus.cc:348] T 2c5c502325fb475f8f050ccad34feb2a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:33.471997 836 raft_consensus.cc:374] T 2c5c502325fb475f8f050ccad34feb2a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.472024 836 raft_consensus.cc:729] T 2c5c502325fb475f8f050ccad34feb2a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.472127 836 consensus_queue.cc:260] T 2c5c502325fb475f8f050ccad34feb2a P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:33.472379 836 ts_tablet_manager.cc:1398] T 2c5c502325fb475f8f050ccad34feb2a P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:33.472525 836 tablet_bootstrap.cc:492] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:33.473227 836 tablet_bootstrap.cc:654] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.474468 32401 tablet_service.cc:1467] Processing CreateTablet for tablet 66471362a1d24a8dab53f9cad512a73a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e53010e4736c42a899edac137f9182cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:33.474622 29994 tablet_service.cc:1467] Processing CreateTablet for tablet 66471362a1d24a8dab53f9cad512a73a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e53010e4736c42a899edac137f9182cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:33.474776 32401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66471362a1d24a8dab53f9cad512a73a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.474866 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66471362a1d24a8dab53f9cad512a73a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.476542 32402 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5c502325fb475f8f050ccad34feb2a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e53010e4736c42a899edac137f9182cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:33.476794 32402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5c502325fb475f8f050ccad34feb2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.477833 842 tablet_bootstrap.cc:492] T 66471362a1d24a8dab53f9cad512a73a P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:33.478008 836 tablet_bootstrap.cc:492] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:33.478082 836 ts_tablet_manager.cc:1367] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:33.478494 836 raft_consensus.cc:348] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:33.478583 836 raft_consensus.cc:374] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.478614 836 raft_consensus.cc:729] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.478740 842 tablet_bootstrap.cc:654] T 66471362a1d24a8dab53f9cad512a73a P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.478744 836 consensus_queue.cc:260] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:33.479014 836 ts_tablet_manager.cc:1398] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:33.480038 842 tablet_bootstrap.cc:492] T 66471362a1d24a8dab53f9cad512a73a P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:33.480051 328 tablet_service.cc:1467] Processing CreateTablet for tablet 66471362a1d24a8dab53f9cad512a73a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=e53010e4736c42a899edac137f9182cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:33.480118 842 ts_tablet_manager.cc:1367] T 66471362a1d24a8dab53f9cad512a73a P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:33.480930 842 raft_consensus.cc:348] T 66471362a1d24a8dab53f9cad512a73a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.481079 842 raft_consensus.cc:374] T 66471362a1d24a8dab53f9cad512a73a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.481139 842 raft_consensus.cc:729] T 66471362a1d24a8dab53f9cad512a73a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.481304 842 consensus_queue.cc:260] T 66471362a1d24a8dab53f9cad512a73a P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.481662 842 ts_tablet_manager.cc:1398] T 66471362a1d24a8dab53f9cad512a73a P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:33.482697 328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66471362a1d24a8dab53f9cad512a73a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.488782 837 tablet_bootstrap.cc:492] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:33.489101 840 tablet_bootstrap.cc:492] T 2c5c502325fb475f8f050ccad34feb2a P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:33.489327 843 tablet_bootstrap.cc:492] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:33.489641 837 tablet_bootstrap.cc:654] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.490371 29867 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:33.490494 29867 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:33.491456 840 tablet_bootstrap.cc:654] T 2c5c502325fb475f8f050ccad34feb2a P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.491761 843 tablet_bootstrap.cc:654] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.491910 837 tablet_bootstrap.cc:492] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:33.492007 837 ts_tablet_manager.cc:1367] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:33.492502 837 raft_consensus.cc:348] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.492602 837 raft_consensus.cc:374] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.492633 837 raft_consensus.cc:729] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.492741 837 consensus_queue.cc:260] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.492954 29867 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 5862d4b94e314e92aeba097f84a46f6a
I20240501 14:09:33.492997 837 ts_tablet_manager.cc:1398] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:33.493922 32665 tablet_service.cc:1514] Processing DeleteTablet for tablet 5862d4b94e314e92aeba097f84a46f6a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:33.494901 32401 tablet_service.cc:1514] Processing DeleteTablet for tablet 5862d4b94e314e92aeba097f84a46f6a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:33.495915 856 tablet_replica.cc:330] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:33.496047 329 tablet_service.cc:1514] Processing DeleteTablet for tablet 5862d4b94e314e92aeba097f84a46f6a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:33.496265 856 raft_consensus.cc:2229] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.496474 856 raft_consensus.cc:2258] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.496627 858 tablet_replica.cc:330] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:33.496747 858 raft_consensus.cc:2229] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:33.497076 858 raft_consensus.cc:2258] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.497223 851 tablet_replica.cc:330] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:33.497478 856 ts_tablet_manager.cc:1875] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.497488 851 raft_consensus.cc:2229] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.497570 843 tablet_bootstrap.cc:492] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:33.497639 858 ts_tablet_manager.cc:1875] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.497670 851 raft_consensus.cc:2258] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.497741 843 ts_tablet_manager.cc:1367] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240501 14:09:33.498646 851 ts_tablet_manager.cc:1875] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.500044 858 ts_tablet_manager.cc:1888] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:33.500147 858 log.cc:1196] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/5862d4b94e314e92aeba097f84a46f6a
I20240501 14:09:33.500630 858 ts_tablet_manager.cc:1909] T 5862d4b94e314e92aeba097f84a46f6a P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:09:33.500896 856 ts_tablet_manager.cc:1888] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:33.501076 856 log.cc:1196] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/5862d4b94e314e92aeba097f84a46f6a
I20240501 14:09:33.501334 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 5862d4b94e314e92aeba097f84a46f6a (table rebalancer_dynamic_table_new_name [id=771f8f7b55af4d5c8613dbcbcf7cf281]) successfully deleted
I20240501 14:09:33.501525 856 ts_tablet_manager.cc:1909] T 5862d4b94e314e92aeba097f84a46f6a P a91029b90bf9415a89a2f70736759ad3: Deleting consensus metadata
I20240501 14:09:33.500880 843 raft_consensus.cc:348] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:33.501895 843 raft_consensus.cc:374] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.501964 843 raft_consensus.cc:729] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.502090 843 consensus_queue.cc:260] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:33.502815 843 ts_tablet_manager.cc:1398] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:33.502976 843 tablet_bootstrap.cc:492] T 66471362a1d24a8dab53f9cad512a73a P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:33.503477 851 ts_tablet_manager.cc:1888] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:33.503587 851 log.cc:1196] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/5862d4b94e314e92aeba097f84a46f6a
I20240501 14:09:33.504478 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 5862d4b94e314e92aeba097f84a46f6a (table rebalancer_dynamic_table_new_name [id=771f8f7b55af4d5c8613dbcbcf7cf281]) successfully deleted
I20240501 14:09:33.505007 843 tablet_bootstrap.cc:654] T 66471362a1d24a8dab53f9cad512a73a P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.505151 851 ts_tablet_manager.cc:1909] T 5862d4b94e314e92aeba097f84a46f6a P db935501e9104de5b15c8780681d6c38: Deleting consensus metadata
I20240501 14:09:33.506258 843 tablet_bootstrap.cc:492] T 66471362a1d24a8dab53f9cad512a73a P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:33.506340 843 ts_tablet_manager.cc:1367] T 66471362a1d24a8dab53f9cad512a73a P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:33.506778 843 raft_consensus.cc:348] T 66471362a1d24a8dab53f9cad512a73a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.506867 843 raft_consensus.cc:374] T 66471362a1d24a8dab53f9cad512a73a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.506897 843 raft_consensus.cc:729] T 66471362a1d24a8dab53f9cad512a73a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.507001 843 consensus_queue.cc:260] T 66471362a1d24a8dab53f9cad512a73a P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.507243 843 ts_tablet_manager.cc:1398] T 66471362a1d24a8dab53f9cad512a73a P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:33.507373 840 tablet_bootstrap.cc:492] T 2c5c502325fb475f8f050ccad34feb2a P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:33.507460 840 ts_tablet_manager.cc:1367] T 2c5c502325fb475f8f050ccad34feb2a P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 14:09:33.507891 840 raft_consensus.cc:348] T 2c5c502325fb475f8f050ccad34feb2a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:33.507990 840 raft_consensus.cc:374] T 2c5c502325fb475f8f050ccad34feb2a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.508033 840 raft_consensus.cc:729] T 2c5c502325fb475f8f050ccad34feb2a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.508150 840 consensus_queue.cc:260] T 2c5c502325fb475f8f050ccad34feb2a P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:33.508419 840 ts_tablet_manager.cc:1398] T 2c5c502325fb475f8f050ccad34feb2a P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:33.508581 840 tablet_bootstrap.cc:492] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:33.508976 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet 5862d4b94e314e92aeba097f84a46f6a (table rebalancer_dynamic_table_new_name [id=771f8f7b55af4d5c8613dbcbcf7cf281]) successfully deleted
I20240501 14:09:33.509770 840 tablet_bootstrap.cc:654] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.518131 840 tablet_bootstrap.cc:492] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:33.518224 840 ts_tablet_manager.cc:1367] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:33.518608 840 raft_consensus.cc:348] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:33.518697 840 raft_consensus.cc:374] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.518728 840 raft_consensus.cc:729] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.518844 840 consensus_queue.cc:260] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:33.519086 840 ts_tablet_manager.cc:1398] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:33.521883 883 ts_tablet_manager.cc:902] T af39c304693645269a6af59190e95cad P 9ecd907cf5634ac98f3f54f3dbec6849: Initiating tablet copy from peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:33.523125 883 tablet_copy_client.cc:308] T af39c304693645269a6af59190e95cad P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.129:41387
I20240501 14:09:33.528533 617 raft_consensus.cc:482] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:33.528647 617 raft_consensus.cc:504] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:33.529027 617 leader_election.cc:288] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:33.529424 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5c502325fb475f8f050ccad34feb2a" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:33.529567 32554 raft_consensus.cc:2454] T 2c5c502325fb475f8f050ccad34feb2a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:33.530020 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5c502325fb475f8f050ccad34feb2a" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:33.530155 32685 raft_consensus.cc:2454] T 2c5c502325fb475f8f050ccad34feb2a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:33.530712 32359 leader_election.cc:302] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:33.531467 617 raft_consensus.cc:2781] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:33.531530 617 raft_consensus.cc:482] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:33.531574 617 raft_consensus.cc:3037] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.532605 617 raft_consensus.cc:504] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:33.532927 617 leader_election.cc:288] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:33.533406 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5c502325fb475f8f050ccad34feb2a" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:33.533511 32685 raft_consensus.cc:3037] T 2c5c502325fb475f8f050ccad34feb2a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.534442 32685 raft_consensus.cc:2454] T 2c5c502325fb475f8f050ccad34feb2a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:33.534792 32356 leader_election.cc:302] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:33.535298 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5c502325fb475f8f050ccad34feb2a" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:33.535413 32554 raft_consensus.cc:3037] T 2c5c502325fb475f8f050ccad34feb2a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.536304 32554 raft_consensus.cc:2454] T 2c5c502325fb475f8f050ccad34feb2a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:33.536756 617 raft_consensus.cc:2781] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:33.536847 617 raft_consensus.cc:686] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:33.536980 617 consensus_queue.cc:237] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:33.541247 30035 tablet_copy_service.cc:140] P 93f2518fbadf4f9991cda93afacb363e: Received BeginTabletCopySession request for tablet af39c304693645269a6af59190e95cad from peer 9ecd907cf5634ac98f3f54f3dbec6849 ({username='slave'} at 127.22.252.136:37957)
I20240501 14:09:33.541375 30035 tablet_copy_service.cc:161] P 93f2518fbadf4f9991cda93afacb363e: Beginning new tablet copy session on tablet af39c304693645269a6af59190e95cad from peer 9ecd907cf5634ac98f3f54f3dbec6849 at {username='slave'} at 127.22.252.136:37957: session id = 9ecd907cf5634ac98f3f54f3dbec6849-af39c304693645269a6af59190e95cad
I20240501 14:09:33.542809 30035 tablet_copy_source_session.cc:205] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:33.540961 29865 catalog_manager.cc:5484] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:33.544672 883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af39c304693645269a6af59190e95cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.546949 883 tablet_copy_client.cc:783] T af39c304693645269a6af59190e95cad P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:33.547119 883 tablet_copy_client.cc:647] T af39c304693645269a6af59190e95cad P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:33.550130 883 tablet_copy_client.cc:523] T af39c304693645269a6af59190e95cad P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:33.553984 883 tablet_bootstrap.cc:492] T af39c304693645269a6af59190e95cad P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:33.554551 890 ts_tablet_manager.cc:902] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3: Initiating tablet copy from peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:33.555702 890 tablet_copy_client.cc:308] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.130:36457
I20240501 14:09:33.560523 30166 tablet_copy_service.cc:140] P d35c16dcb506426993811bb2519d7db3: Received BeginTabletCopySession request for tablet 796cef2485c94b76a5f3e1944b73dcf6 from peer a91029b90bf9415a89a2f70736759ad3 ({username='slave'} at 127.22.252.133:37313)
I20240501 14:09:33.560631 30166 tablet_copy_service.cc:161] P d35c16dcb506426993811bb2519d7db3: Beginning new tablet copy session on tablet 796cef2485c94b76a5f3e1944b73dcf6 from peer a91029b90bf9415a89a2f70736759ad3 at {username='slave'} at 127.22.252.133:37313: session id = a91029b90bf9415a89a2f70736759ad3-796cef2485c94b76a5f3e1944b73dcf6
I20240501 14:09:33.561789 30166 tablet_copy_source_session.cc:205] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:33.563213 890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 796cef2485c94b76a5f3e1944b73dcf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.566882 890 tablet_copy_client.cc:783] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:33.567060 890 tablet_copy_client.cc:647] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:33.569568 890 tablet_copy_client.cc:523] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:33.574434 890 tablet_bootstrap.cc:492] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:33.593729 883 tablet_bootstrap.cc:492] T af39c304693645269a6af59190e95cad P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:33.594110 883 tablet_bootstrap.cc:492] T af39c304693645269a6af59190e95cad P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap complete.
I20240501 14:09:33.594301 883 ts_tablet_manager.cc:1367] T af39c304693645269a6af59190e95cad P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.040s user 0.030s sys 0.000s
I20240501 14:09:33.594733 883 raft_consensus.cc:348] T af39c304693645269a6af59190e95cad P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.594882 883 raft_consensus.cc:729] T af39c304693645269a6af59190e95cad P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.595048 883 consensus_queue.cc:260] T af39c304693645269a6af59190e95cad P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.596410 883 ts_tablet_manager.cc:1398] T af39c304693645269a6af59190e95cad P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:33.597069 30035 tablet_copy_service.cc:342] P 93f2518fbadf4f9991cda93afacb363e: Request end of tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-af39c304693645269a6af59190e95cad received from {username='slave'} at 127.22.252.136:37957
I20240501 14:09:33.597155 30035 tablet_copy_service.cc:433] P 93f2518fbadf4f9991cda93afacb363e: ending tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-af39c304693645269a6af59190e95cad on tablet af39c304693645269a6af59190e95cad with peer 9ecd907cf5634ac98f3f54f3dbec6849
I20240501 14:09:33.599555 29870 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:33.606773 890 tablet_bootstrap.cc:492] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:33.607050 890 tablet_bootstrap.cc:492] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3: Bootstrap complete.
I20240501 14:09:33.607236 890 ts_tablet_manager.cc:1367] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.004s
I20240501 14:09:33.607581 890 raft_consensus.cc:348] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:33.607676 890 raft_consensus.cc:729] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.607823 890 consensus_queue.cc:260] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:33.608117 890 ts_tablet_manager.cc:1398] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:33.608721 30166 tablet_copy_service.cc:342] P d35c16dcb506426993811bb2519d7db3: Request end of tablet copy session a91029b90bf9415a89a2f70736759ad3-796cef2485c94b76a5f3e1944b73dcf6 received from {username='slave'} at 127.22.252.133:37313
I20240501 14:09:33.608819 30166 tablet_copy_service.cc:433] P d35c16dcb506426993811bb2519d7db3: ending tablet copy session a91029b90bf9415a89a2f70736759ad3-796cef2485c94b76a5f3e1944b73dcf6 on tablet 796cef2485c94b76a5f3e1944b73dcf6 with peer a91029b90bf9415a89a2f70736759ad3
I20240501 14:09:33.615206 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 76f2a93fab2d4c668365be3335d4b1ad (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c4d4abe1d4e4474e8ee4f5b565b937a9]), partition=
I20240501 14:09:33.615430 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f2a93fab2d4c668365be3335d4b1ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.616791 329 tablet_service.cc:1467] Processing CreateTablet for tablet 76f2a93fab2d4c668365be3335d4b1ad (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c4d4abe1d4e4474e8ee4f5b565b937a9]), partition=
I20240501 14:09:33.617045 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f2a93fab2d4c668365be3335d4b1ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.619042 837 tablet_bootstrap.cc:492] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:33.619411 898 tablet_bootstrap.cc:492] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:33.620280 898 tablet_bootstrap.cc:654] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.620476 837 tablet_bootstrap.cc:654] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.623306 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 76f2a93fab2d4c668365be3335d4b1ad (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c4d4abe1d4e4474e8ee4f5b565b937a9]), partition=
I20240501 14:09:33.623530 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f2a93fab2d4c668365be3335d4b1ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.625473 836 tablet_bootstrap.cc:492] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:33.626286 836 tablet_bootstrap.cc:654] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.627310 836 tablet_bootstrap.cc:492] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:33.627388 836 ts_tablet_manager.cc:1367] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:33.627777 836 raft_consensus.cc:348] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.627877 836 raft_consensus.cc:374] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.627913 836 raft_consensus.cc:729] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.628037 836 consensus_queue.cc:260] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.628312 836 ts_tablet_manager.cc:1398] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:33.628801 837 tablet_bootstrap.cc:492] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:33.628886 837 ts_tablet_manager.cc:1367] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20240501 14:09:33.629352 837 raft_consensus.cc:348] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.629438 837 raft_consensus.cc:374] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.629463 837 raft_consensus.cc:729] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.629559 837 consensus_queue.cc:260] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.629771 837 ts_tablet_manager.cc:1398] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:33.632221 898 tablet_bootstrap.cc:492] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:33.632462 898 ts_tablet_manager.cc:1367] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240501 14:09:33.633016 898 raft_consensus.cc:348] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.633141 898 raft_consensus.cc:374] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.633199 898 raft_consensus.cc:729] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.633371 898 consensus_queue.cc:260] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.633702 898 ts_tablet_manager.cc:1398] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:33.657516 902 ts_tablet_manager.cc:902] T dd2fd47ec6684cb28d2f4678d2e90e75 P db935501e9104de5b15c8780681d6c38: Initiating tablet copy from peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:33.658320 902 tablet_copy_client.cc:308] T dd2fd47ec6684cb28d2f4678d2e90e75 P db935501e9104de5b15c8780681d6c38: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.132:34357
I20240501 14:09:33.661855 30428 tablet_copy_service.cc:140] P d1c217469ae4478ebd49e4ed9f7352d7: Received BeginTabletCopySession request for tablet dd2fd47ec6684cb28d2f4678d2e90e75 from peer db935501e9104de5b15c8780681d6c38 ({username='slave'} at 127.22.252.135:43131)
I20240501 14:09:33.661998 30428 tablet_copy_service.cc:161] P d1c217469ae4478ebd49e4ed9f7352d7: Beginning new tablet copy session on tablet dd2fd47ec6684cb28d2f4678d2e90e75 from peer db935501e9104de5b15c8780681d6c38 at {username='slave'} at 127.22.252.135:43131: session id = db935501e9104de5b15c8780681d6c38-dd2fd47ec6684cb28d2f4678d2e90e75
I20240501 14:09:33.661979 29873 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 14:09:33.662122 29873 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 14:09:33.663264 30428 tablet_copy_source_session.cc:205] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:33.664567 29873 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet ef6731d4b3824d3eaeba37e2fb464d32
I20240501 14:09:33.664840 29873 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 6f3e51eb67b947038487dc0c7782d0b6
I20240501 14:09:33.665117 29873 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 15f40a19911b40fb980eead07bcbd106
I20240501 14:09:33.665132 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet ef6731d4b3824d3eaeba37e2fb464d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:33.665208 30256 tablet_service.cc:1514] Processing DeleteTablet for tablet ef6731d4b3824d3eaeba37e2fb464d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:33.665185 29994 tablet_service.cc:1514] Processing DeleteTablet for tablet ef6731d4b3824d3eaeba37e2fb464d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:33.665297 543 tablet_replica.cc:330] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:33.665374 545 tablet_replica.cc:330] T ef6731d4b3824d3eaeba37e2fb464d32 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:33.665446 543 raft_consensus.cc:2229] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:33.665524 545 raft_consensus.cc:2229] T ef6731d4b3824d3eaeba37e2fb464d32 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.665632 543 raft_consensus.cc:2258] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.665675 545 raft_consensus.cc:2258] T ef6731d4b3824d3eaeba37e2fb464d32 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.666105 902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd2fd47ec6684cb28d2f4678d2e90e75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.666121 545 ts_tablet_manager.cc:1875] T ef6731d4b3824d3eaeba37e2fb464d32 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.666143 543 ts_tablet_manager.cc:1875] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.667894 545 ts_tablet_manager.cc:1888] T ef6731d4b3824d3eaeba37e2fb464d32 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.667963 545 log.cc:1196] T ef6731d4b3824d3eaeba37e2fb464d32 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/ef6731d4b3824d3eaeba37e2fb464d32
I20240501 14:09:33.668311 545 ts_tablet_manager.cc:1909] T ef6731d4b3824d3eaeba37e2fb464d32 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:33.668354 544 tablet_replica.cc:330] T ef6731d4b3824d3eaeba37e2fb464d32 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:33.668395 29995 tablet_service.cc:1514] Processing DeleteTablet for tablet 15f40a19911b40fb980eead07bcbd106 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:33.668428 902 tablet_copy_client.cc:783] T dd2fd47ec6684cb28d2f4678d2e90e75 P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:33.668498 544 raft_consensus.cc:2229] T ef6731d4b3824d3eaeba37e2fb464d32 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.668584 902 tablet_copy_client.cc:647] T dd2fd47ec6684cb28d2f4678d2e90e75 P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:33.668607 29994 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f3e51eb67b947038487dc0c7782d0b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:33.668668 544 raft_consensus.cc:2258] T ef6731d4b3824d3eaeba37e2fb464d32 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.668915 30256 tablet_service.cc:1514] Processing DeleteTablet for tablet 15f40a19911b40fb980eead07bcbd106 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:33.669044 545 tablet_replica.cc:330] T 15f40a19911b40fb980eead07bcbd106 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:33.669144 545 raft_consensus.cc:2229] T 15f40a19911b40fb980eead07bcbd106 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.669160 544 ts_tablet_manager.cc:1875] T ef6731d4b3824d3eaeba37e2fb464d32 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.669274 545 raft_consensus.cc:2258] T 15f40a19911b40fb980eead07bcbd106 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.669339 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet ef6731d4b3824d3eaeba37e2fb464d32 (table rebalance_test_table_4 [id=979bbb7a065445c89d4cf2f3e28ed0d9]) successfully deleted
I20240501 14:09:33.669759 545 ts_tablet_manager.cc:1875] T 15f40a19911b40fb980eead07bcbd106 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.670792 544 ts_tablet_manager.cc:1888] T ef6731d4b3824d3eaeba37e2fb464d32 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.670840 544 log.cc:1196] T ef6731d4b3824d3eaeba37e2fb464d32 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/ef6731d4b3824d3eaeba37e2fb464d32
I20240501 14:09:33.671032 544 ts_tablet_manager.cc:1909] T ef6731d4b3824d3eaeba37e2fb464d32 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting consensus metadata
I20240501 14:09:33.671330 545 ts_tablet_manager.cc:1888] T 15f40a19911b40fb980eead07bcbd106 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.671389 545 log.cc:1196] T 15f40a19911b40fb980eead07bcbd106 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/15f40a19911b40fb980eead07bcbd106
I20240501 14:09:33.671501 30387 tablet_service.cc:1514] Processing DeleteTablet for tablet 15f40a19911b40fb980eead07bcbd106 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:33.671666 546 tablet_replica.cc:330] T 15f40a19911b40fb980eead07bcbd106 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:33.671805 546 raft_consensus.cc:2229] T 15f40a19911b40fb980eead07bcbd106 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.671838 30388 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f3e51eb67b947038487dc0c7782d0b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:33.671905 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f3e51eb67b947038487dc0c7782d0b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:33.671700 544 tablet_replica.cc:330] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:33.671947 546 raft_consensus.cc:2258] T 15f40a19911b40fb980eead07bcbd106 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.672042 544 raft_consensus.cc:2229] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:33.672228 544 raft_consensus.cc:2258] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.672403 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet ef6731d4b3824d3eaeba37e2fb464d32 (table rebalance_test_table_4 [id=979bbb7a065445c89d4cf2f3e28ed0d9]) successfully deleted
I20240501 14:09:33.672474 546 ts_tablet_manager.cc:1875] T 15f40a19911b40fb980eead07bcbd106 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.672513 543 ts_tablet_manager.cc:1888] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.672577 543 log.cc:1196] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/ef6731d4b3824d3eaeba37e2fb464d32
I20240501 14:09:33.672816 544 ts_tablet_manager.cc:1875] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.672847 543 ts_tablet_manager.cc:1909] T ef6731d4b3824d3eaeba37e2fb464d32 P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:33.673372 543 tablet_replica.cc:330] T 6f3e51eb67b947038487dc0c7782d0b6 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:33.673485 543 raft_consensus.cc:2229] T 6f3e51eb67b947038487dc0c7782d0b6 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.673501 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet ef6731d4b3824d3eaeba37e2fb464d32 (table rebalance_test_table_4 [id=979bbb7a065445c89d4cf2f3e28ed0d9]) successfully deleted
I20240501 14:09:33.673609 543 raft_consensus.cc:2258] T 6f3e51eb67b947038487dc0c7782d0b6 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.674055 543 ts_tablet_manager.cc:1875] T 6f3e51eb67b947038487dc0c7782d0b6 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.674289 546 ts_tablet_manager.cc:1888] T 15f40a19911b40fb980eead07bcbd106 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.674350 546 log.cc:1196] T 15f40a19911b40fb980eead07bcbd106 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/15f40a19911b40fb980eead07bcbd106
I20240501 14:09:33.674537 546 ts_tablet_manager.cc:1909] T 15f40a19911b40fb980eead07bcbd106 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:33.674546 544 ts_tablet_manager.cc:1888] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.674603 544 log.cc:1196] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/15f40a19911b40fb980eead07bcbd106
I20240501 14:09:33.674757 545 ts_tablet_manager.cc:1909] T 15f40a19911b40fb980eead07bcbd106 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:33.674875 544 ts_tablet_manager.cc:1909] T 15f40a19911b40fb980eead07bcbd106 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting consensus metadata
I20240501 14:09:33.675396 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 15f40a19911b40fb980eead07bcbd106 (table rebalance_test_table_4 [id=979bbb7a065445c89d4cf2f3e28ed0d9]) successfully deleted
I20240501 14:09:33.675431 546 tablet_replica.cc:330] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:33.675558 546 raft_consensus.cc:2229] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:33.675588 545 tablet_replica.cc:330] T 6f3e51eb67b947038487dc0c7782d0b6 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:33.675705 545 raft_consensus.cc:2229] T 6f3e51eb67b947038487dc0c7782d0b6 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.675889 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 15f40a19911b40fb980eead07bcbd106 (table rebalance_test_table_4 [id=979bbb7a065445c89d4cf2f3e28ed0d9]) successfully deleted
I20240501 14:09:33.675927 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet 15f40a19911b40fb980eead07bcbd106 (table rebalance_test_table_4 [id=979bbb7a065445c89d4cf2f3e28ed0d9]) successfully deleted
I20240501 14:09:33.676038 545 raft_consensus.cc:2258] T 6f3e51eb67b947038487dc0c7782d0b6 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.676195 546 raft_consensus.cc:2258] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.676560 545 ts_tablet_manager.cc:1875] T 6f3e51eb67b947038487dc0c7782d0b6 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.676632 546 ts_tablet_manager.cc:1875] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.677203 543 ts_tablet_manager.cc:1888] T 6f3e51eb67b947038487dc0c7782d0b6 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.677251 543 log.cc:1196] T 6f3e51eb67b947038487dc0c7782d0b6 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/6f3e51eb67b947038487dc0c7782d0b6
I20240501 14:09:33.677428 543 ts_tablet_manager.cc:1909] T 6f3e51eb67b947038487dc0c7782d0b6 P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:33.677803 902 tablet_copy_client.cc:523] T dd2fd47ec6684cb28d2f4678d2e90e75 P db935501e9104de5b15c8780681d6c38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:33.678345 545 ts_tablet_manager.cc:1888] T 6f3e51eb67b947038487dc0c7782d0b6 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.678345 546 ts_tablet_manager.cc:1888] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.678406 546 log.cc:1196] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/6f3e51eb67b947038487dc0c7782d0b6
I20240501 14:09:33.678406 545 log.cc:1196] T 6f3e51eb67b947038487dc0c7782d0b6 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/6f3e51eb67b947038487dc0c7782d0b6
I20240501 14:09:33.678655 545 ts_tablet_manager.cc:1909] T 6f3e51eb67b947038487dc0c7782d0b6 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:33.678656 546 ts_tablet_manager.cc:1909] T 6f3e51eb67b947038487dc0c7782d0b6 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:33.679347 902 tablet_bootstrap.cc:492] T dd2fd47ec6684cb28d2f4678d2e90e75 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:33.679327 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 6f3e51eb67b947038487dc0c7782d0b6 (table rebalance_test_table_4 [id=979bbb7a065445c89d4cf2f3e28ed0d9]) successfully deleted
I20240501 14:09:33.679347 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 6f3e51eb67b947038487dc0c7782d0b6 (table rebalance_test_table_4 [id=979bbb7a065445c89d4cf2f3e28ed0d9]) successfully deleted
I20240501 14:09:33.679347 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 6f3e51eb67b947038487dc0c7782d0b6 (table rebalance_test_table_4 [id=979bbb7a065445c89d4cf2f3e28ed0d9]) successfully deleted
I20240501 14:09:33.689973 909 ts_tablet_manager.cc:902] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 5f8a0cfdfdca4148a75686dc88059cf3: Initiating tablet copy from peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:33.690790 909 tablet_copy_client.cc:308] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.130:36457
I20240501 14:09:33.694955 30166 tablet_copy_service.cc:140] P d35c16dcb506426993811bb2519d7db3: Received BeginTabletCopySession request for tablet 8d5e1afe21654080b5aa3eb877e8f1d9 from peer 5f8a0cfdfdca4148a75686dc88059cf3 ({username='slave'} at 127.22.252.134:40399)
I20240501 14:09:33.695052 30166 tablet_copy_service.cc:161] P d35c16dcb506426993811bb2519d7db3: Beginning new tablet copy session on tablet 8d5e1afe21654080b5aa3eb877e8f1d9 from peer 5f8a0cfdfdca4148a75686dc88059cf3 at {username='slave'} at 127.22.252.134:40399: session id = 5f8a0cfdfdca4148a75686dc88059cf3-8d5e1afe21654080b5aa3eb877e8f1d9
I20240501 14:09:33.695766 30166 tablet_copy_source_session.cc:205] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:33.696975 909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5e1afe21654080b5aa3eb877e8f1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.699259 909 tablet_copy_client.cc:783] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:33.699507 909 tablet_copy_client.cc:647] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:33.701049 909 tablet_copy_client.cc:523] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:33.702564 909 tablet_bootstrap.cc:492] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:33.703688 902 tablet_bootstrap.cc:492] T dd2fd47ec6684cb28d2f4678d2e90e75 P db935501e9104de5b15c8780681d6c38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:33.703972 902 tablet_bootstrap.cc:492] T dd2fd47ec6684cb28d2f4678d2e90e75 P db935501e9104de5b15c8780681d6c38: Bootstrap complete.
I20240501 14:09:33.704131 902 ts_tablet_manager.cc:1367] T dd2fd47ec6684cb28d2f4678d2e90e75 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.025s user 0.018s sys 0.004s
I20240501 14:09:33.704602 902 raft_consensus.cc:348] T dd2fd47ec6684cb28d2f4678d2e90e75 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:33.704726 902 raft_consensus.cc:729] T dd2fd47ec6684cb28d2f4678d2e90e75 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.704867 902 consensus_queue.cc:260] T dd2fd47ec6684cb28d2f4678d2e90e75 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:33.705247 902 ts_tablet_manager.cc:1398] T dd2fd47ec6684cb28d2f4678d2e90e75 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240501 14:09:33.705847 30428 tablet_copy_service.cc:342] P d1c217469ae4478ebd49e4ed9f7352d7: Request end of tablet copy session db935501e9104de5b15c8780681d6c38-dd2fd47ec6684cb28d2f4678d2e90e75 received from {username='slave'} at 127.22.252.135:43131
I20240501 14:09:33.705943 30428 tablet_copy_service.cc:433] P d1c217469ae4478ebd49e4ed9f7352d7: ending tablet copy session db935501e9104de5b15c8780681d6c38-dd2fd47ec6684cb28d2f4678d2e90e75 on tablet dd2fd47ec6684cb28d2f4678d2e90e75 with peer db935501e9104de5b15c8780681d6c38
I20240501 14:09:33.724779 909 tablet_bootstrap.cc:492] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:33.725116 909 tablet_bootstrap.cc:492] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap complete.
I20240501 14:09:33.725337 909 ts_tablet_manager.cc:1367] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.009s
I20240501 14:09:33.725764 909 raft_consensus.cc:348] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:33.725864 909 raft_consensus.cc:729] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.725999 909 consensus_queue.cc:260] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:33.726336 909 ts_tablet_manager.cc:1398] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:33.726787 30166 tablet_copy_service.cc:342] P d35c16dcb506426993811bb2519d7db3: Request end of tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-8d5e1afe21654080b5aa3eb877e8f1d9 received from {username='slave'} at 127.22.252.134:40399
I20240501 14:09:33.726862 30166 tablet_copy_service.cc:433] P d35c16dcb506426993811bb2519d7db3: ending tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-8d5e1afe21654080b5aa3eb877e8f1d9 on tablet 8d5e1afe21654080b5aa3eb877e8f1d9 with peer 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:09:33.735253 640 raft_consensus.cc:482] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:33.735385 640 raft_consensus.cc:504] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.735806 640 leader_election.cc:288] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:33.736181 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66471362a1d24a8dab53f9cad512a73a" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:33.736171 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66471362a1d24a8dab53f9cad512a73a" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:33.736349 30014 raft_consensus.cc:2454] T 66471362a1d24a8dab53f9cad512a73a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:33.736361 32415 raft_consensus.cc:2454] T 66471362a1d24a8dab53f9cad512a73a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:33.736649 32752 leader_election.cc:302] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:33.736807 640 raft_consensus.cc:2781] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:33.736863 640 raft_consensus.cc:482] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:33.736912 640 raft_consensus.cc:3037] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.737684 640 raft_consensus.cc:504] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.737962 640 leader_election.cc:288] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:33.738332 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66471362a1d24a8dab53f9cad512a73a" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:33.738332 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66471362a1d24a8dab53f9cad512a73a" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:33.738445 32415 raft_consensus.cc:3037] T 66471362a1d24a8dab53f9cad512a73a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.738445 30014 raft_consensus.cc:3037] T 66471362a1d24a8dab53f9cad512a73a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.739403 32415 raft_consensus.cc:2454] T 66471362a1d24a8dab53f9cad512a73a P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:33.739415 30014 raft_consensus.cc:2454] T 66471362a1d24a8dab53f9cad512a73a P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:33.739708 32752 leader_election.cc:302] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:33.739851 640 raft_consensus.cc:2781] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:33.739945 640 raft_consensus.cc:686] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:33.740089 640 consensus_queue.cc:237] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.741595 29870 catalog_manager.cc:5484] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:33.773213 566 raft_consensus.cc:482] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:33.773325 566 raft_consensus.cc:504] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.774338 566 leader_election.cc:288] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:33.775658 890 ts_tablet_manager.cc:902] T 4f5314fbd59c4443974a9c304ba9b1a7 P a91029b90bf9415a89a2f70736759ad3: Initiating tablet copy from peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:33.776116 890 tablet_copy_client.cc:308] T 4f5314fbd59c4443974a9c304ba9b1a7 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.132:34357
I20240501 14:09:33.778326 30408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f2a93fab2d4c668365be3335d4b1ad" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:33.778460 30408 raft_consensus.cc:2454] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:33.778808 32487 leader_election.cc:302] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:33.779062 566 raft_consensus.cc:2781] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:33.779124 566 raft_consensus.cc:482] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:33.779156 566 raft_consensus.cc:3037] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.779220 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f2a93fab2d4c668365be3335d4b1ad" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:33.779382 349 raft_consensus.cc:2454] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:33.780278 566 raft_consensus.cc:504] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.780649 566 leader_election.cc:288] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:33.780879 30408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f2a93fab2d4c668365be3335d4b1ad" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:33.780969 30428 tablet_copy_service.cc:140] P d1c217469ae4478ebd49e4ed9f7352d7: Received BeginTabletCopySession request for tablet 4f5314fbd59c4443974a9c304ba9b1a7 from peer a91029b90bf9415a89a2f70736759ad3 ({username='slave'} at 127.22.252.133:43857)
I20240501 14:09:33.780994 30408 raft_consensus.cc:3037] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.781069 30428 tablet_copy_service.cc:161] P d1c217469ae4478ebd49e4ed9f7352d7: Beginning new tablet copy session on tablet 4f5314fbd59c4443974a9c304ba9b1a7 from peer a91029b90bf9415a89a2f70736759ad3 at {username='slave'} at 127.22.252.133:43857: session id = a91029b90bf9415a89a2f70736759ad3-4f5314fbd59c4443974a9c304ba9b1a7
I20240501 14:09:33.781841 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f2a93fab2d4c668365be3335d4b1ad" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:33.781963 349 raft_consensus.cc:3037] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.781994 30408 raft_consensus.cc:2454] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:33.782078 30428 tablet_copy_source_session.cc:205] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:33.782243 32487 leader_election.cc:302] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:33.782418 566 raft_consensus.cc:2781] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:33.782572 566 raft_consensus.cc:686] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:09:33.782879 566 consensus_queue.cc:237] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.782994 349 raft_consensus.cc:2454] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:33.783502 890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f5314fbd59c4443974a9c304ba9b1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.785146 29870 catalog_manager.cc:5484] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:33.785530 890 tablet_copy_client.cc:783] T 4f5314fbd59c4443974a9c304ba9b1a7 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:33.785668 890 tablet_copy_client.cc:647] T 4f5314fbd59c4443974a9c304ba9b1a7 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:33.787297 890 tablet_copy_client.cc:523] T 4f5314fbd59c4443974a9c304ba9b1a7 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:33.788367 890 tablet_bootstrap.cc:492] T 4f5314fbd59c4443974a9c304ba9b1a7 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:33.811383 890 tablet_bootstrap.cc:492] T 4f5314fbd59c4443974a9c304ba9b1a7 P a91029b90bf9415a89a2f70736759ad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:33.811636 890 tablet_bootstrap.cc:492] T 4f5314fbd59c4443974a9c304ba9b1a7 P a91029b90bf9415a89a2f70736759ad3: Bootstrap complete.
I20240501 14:09:33.811801 890 ts_tablet_manager.cc:1367] T 4f5314fbd59c4443974a9c304ba9b1a7 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.000s
I20240501 14:09:33.812209 890 raft_consensus.cc:348] T 4f5314fbd59c4443974a9c304ba9b1a7 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:33.812381 890 raft_consensus.cc:729] T 4f5314fbd59c4443974a9c304ba9b1a7 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.812521 890 consensus_queue.cc:260] T 4f5314fbd59c4443974a9c304ba9b1a7 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:33.812783 890 ts_tablet_manager.cc:1398] T 4f5314fbd59c4443974a9c304ba9b1a7 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 14:09:33.813274 30428 tablet_copy_service.cc:342] P d1c217469ae4478ebd49e4ed9f7352d7: Request end of tablet copy session a91029b90bf9415a89a2f70736759ad3-4f5314fbd59c4443974a9c304ba9b1a7 received from {username='slave'} at 127.22.252.133:43857
I20240501 14:09:33.813346 30428 tablet_copy_service.cc:433] P d1c217469ae4478ebd49e4ed9f7352d7: ending tablet copy session a91029b90bf9415a89a2f70736759ad3-4f5314fbd59c4443974a9c304ba9b1a7 on tablet 4f5314fbd59c4443974a9c304ba9b1a7 with peer a91029b90bf9415a89a2f70736759ad3
I20240501 14:09:33.833657 909 ts_tablet_manager.cc:902] T 5da2beba68ba414281025fcee91ca0d0 P 5f8a0cfdfdca4148a75686dc88059cf3: Initiating tablet copy from peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:33.834211 909 tablet_copy_client.cc:308] T 5da2beba68ba414281025fcee91ca0d0 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.129:41387
I20240501 14:09:33.837159 30035 tablet_copy_service.cc:140] P 93f2518fbadf4f9991cda93afacb363e: Received BeginTabletCopySession request for tablet 5da2beba68ba414281025fcee91ca0d0 from peer 5f8a0cfdfdca4148a75686dc88059cf3 ({username='slave'} at 127.22.252.134:56355)
I20240501 14:09:33.837252 30035 tablet_copy_service.cc:161] P 93f2518fbadf4f9991cda93afacb363e: Beginning new tablet copy session on tablet 5da2beba68ba414281025fcee91ca0d0 from peer 5f8a0cfdfdca4148a75686dc88059cf3 at {username='slave'} at 127.22.252.134:56355: session id = 5f8a0cfdfdca4148a75686dc88059cf3-5da2beba68ba414281025fcee91ca0d0
I20240501 14:09:33.837989 30035 tablet_copy_source_session.cc:205] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:33.838730 909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5da2beba68ba414281025fcee91ca0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.840438 909 tablet_copy_client.cc:783] T 5da2beba68ba414281025fcee91ca0d0 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:33.840606 909 tablet_copy_client.cc:647] T 5da2beba68ba414281025fcee91ca0d0 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:33.841562 909 tablet_copy_client.cc:523] T 5da2beba68ba414281025fcee91ca0d0 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:33.842562 909 tablet_bootstrap.cc:492] T 5da2beba68ba414281025fcee91ca0d0 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:33.847663 883 ts_tablet_manager.cc:902] T 8d87b2a4f6764e20afbfb43681f58467 P 9ecd907cf5634ac98f3f54f3dbec6849: Initiating tablet copy from peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:33.848148 883 tablet_copy_client.cc:308] T 8d87b2a4f6764e20afbfb43681f58467 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.130:36457
I20240501 14:09:33.852073 30166 tablet_copy_service.cc:140] P d35c16dcb506426993811bb2519d7db3: Received BeginTabletCopySession request for tablet 8d87b2a4f6764e20afbfb43681f58467 from peer 9ecd907cf5634ac98f3f54f3dbec6849 ({username='slave'} at 127.22.252.136:49091)
I20240501 14:09:33.852159 30166 tablet_copy_service.cc:161] P d35c16dcb506426993811bb2519d7db3: Beginning new tablet copy session on tablet 8d87b2a4f6764e20afbfb43681f58467 from peer 9ecd907cf5634ac98f3f54f3dbec6849 at {username='slave'} at 127.22.252.136:49091: session id = 9ecd907cf5634ac98f3f54f3dbec6849-8d87b2a4f6764e20afbfb43681f58467
I20240501 14:09:33.852855 30166 tablet_copy_source_session.cc:205] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:33.853638 883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d87b2a4f6764e20afbfb43681f58467. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.855721 883 tablet_copy_client.cc:783] T 8d87b2a4f6764e20afbfb43681f58467 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:33.855859 883 tablet_copy_client.cc:647] T 8d87b2a4f6764e20afbfb43681f58467 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:33.857774 883 tablet_copy_client.cc:523] T 8d87b2a4f6764e20afbfb43681f58467 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:33.858707 883 tablet_bootstrap.cc:492] T 8d87b2a4f6764e20afbfb43681f58467 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:33.864682 909 tablet_bootstrap.cc:492] T 5da2beba68ba414281025fcee91ca0d0 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:33.864908 909 tablet_bootstrap.cc:492] T 5da2beba68ba414281025fcee91ca0d0 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap complete.
I20240501 14:09:33.865061 909 ts_tablet_manager.cc:1367] T 5da2beba68ba414281025fcee91ca0d0 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.023s user 0.022s sys 0.000s
I20240501 14:09:33.865468 909 raft_consensus.cc:348] T 5da2beba68ba414281025fcee91ca0d0 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:33.865618 909 raft_consensus.cc:729] T 5da2beba68ba414281025fcee91ca0d0 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.865785 909 consensus_queue.cc:260] T 5da2beba68ba414281025fcee91ca0d0 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:33.866210 909 ts_tablet_manager.cc:1398] T 5da2beba68ba414281025fcee91ca0d0 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:33.866629 30035 tablet_copy_service.cc:342] P 93f2518fbadf4f9991cda93afacb363e: Request end of tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-5da2beba68ba414281025fcee91ca0d0 received from {username='slave'} at 127.22.252.134:56355
I20240501 14:09:33.866725 30035 tablet_copy_service.cc:433] P 93f2518fbadf4f9991cda93afacb363e: ending tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-5da2beba68ba414281025fcee91ca0d0 on tablet 5da2beba68ba414281025fcee91ca0d0 with peer 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:09:33.878125 883 tablet_bootstrap.cc:492] T 8d87b2a4f6764e20afbfb43681f58467 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:33.878381 883 tablet_bootstrap.cc:492] T 8d87b2a4f6764e20afbfb43681f58467 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap complete.
I20240501 14:09:33.878520 883 ts_tablet_manager.cc:1367] T 8d87b2a4f6764e20afbfb43681f58467 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20240501 14:09:33.878839 883 raft_consensus.cc:348] T 8d87b2a4f6764e20afbfb43681f58467 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.878924 883 raft_consensus.cc:729] T 8d87b2a4f6764e20afbfb43681f58467 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.879050 883 consensus_queue.cc:260] T 8d87b2a4f6764e20afbfb43681f58467 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.879303 883 ts_tablet_manager.cc:1398] T 8d87b2a4f6764e20afbfb43681f58467 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 14:09:33.879760 30166 tablet_copy_service.cc:342] P d35c16dcb506426993811bb2519d7db3: Request end of tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-8d87b2a4f6764e20afbfb43681f58467 received from {username='slave'} at 127.22.252.136:49091
I20240501 14:09:33.879833 30166 tablet_copy_service.cc:433] P d35c16dcb506426993811bb2519d7db3: ending tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-8d87b2a4f6764e20afbfb43681f58467 on tablet 8d87b2a4f6764e20afbfb43681f58467 with peer 9ecd907cf5634ac98f3f54f3dbec6849
I20240501 14:09:33.893590 883 ts_tablet_manager.cc:902] T 940c8f54339f4a32be2942682a41c685 P 9ecd907cf5634ac98f3f54f3dbec6849: Initiating tablet copy from peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:33.893942 883 tablet_copy_client.cc:308] T 940c8f54339f4a32be2942682a41c685 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.132:34357
I20240501 14:09:33.896929 30428 tablet_copy_service.cc:140] P d1c217469ae4478ebd49e4ed9f7352d7: Received BeginTabletCopySession request for tablet 940c8f54339f4a32be2942682a41c685 from peer 9ecd907cf5634ac98f3f54f3dbec6849 ({username='slave'} at 127.22.252.136:38643)
I20240501 14:09:33.897038 30428 tablet_copy_service.cc:161] P d1c217469ae4478ebd49e4ed9f7352d7: Beginning new tablet copy session on tablet 940c8f54339f4a32be2942682a41c685 from peer 9ecd907cf5634ac98f3f54f3dbec6849 at {username='slave'} at 127.22.252.136:38643: session id = 9ecd907cf5634ac98f3f54f3dbec6849-940c8f54339f4a32be2942682a41c685
I20240501 14:09:33.897740 30428 tablet_copy_source_session.cc:205] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:33.898428 883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940c8f54339f4a32be2942682a41c685. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.900069 883 tablet_copy_client.cc:783] T 940c8f54339f4a32be2942682a41c685 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:33.900199 883 tablet_copy_client.cc:647] T 940c8f54339f4a32be2942682a41c685 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:33.900776 29870 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:33.901410 883 tablet_copy_client.cc:523] T 940c8f54339f4a32be2942682a41c685 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:33.902415 883 tablet_bootstrap.cc:492] T 940c8f54339f4a32be2942682a41c685 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:33.905687 349 raft_consensus.cc:1261] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:33.906057 30408 raft_consensus.cc:1261] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:33.906314 566 consensus_queue.cc:1035] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:33.906596 923 consensus_queue.cc:1035] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:33.908841 29870 catalog_manager.cc:5576] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937) does not have the latest schema for tablet 76f2a93fab2d4c668365be3335d4b1ad (table rebalancer_dynamic_table [id=c4d4abe1d4e4474e8ee4f5b565b937a9]). Expected version 1 got 0
I20240501 14:09:33.910365 947 tablet.cc:1722] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:33.910408 29870 catalog_manager.cc:5576] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357) does not have the latest schema for tablet 76f2a93fab2d4c668365be3335d4b1ad (table rebalancer_dynamic_table [id=c4d4abe1d4e4474e8ee4f5b565b937a9]). Expected version 1 got 0
I20240501 14:09:33.910732 947 tablet.cc:1620] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:33.911082 940 mvcc.cc:205] Tried to move back new op lower bound from 7022889262712991744 to 7022889262217142272. Current Snapshot: MvccSnapshot[applied={T|T < 7022889262712991744}]
I20240501 14:09:33.911480 946 tablet.cc:1722] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:33.912104 946 tablet.cc:1620] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:33.912791 566 raft_consensus.cc:482] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:33.912863 566 raft_consensus.cc:504] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:33.913290 566 leader_election.cc:288] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:33.913357 29873 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=c4d4abe1d4e4474e8ee4f5b565b937a9] alter complete (version 1)
I20240501 14:09:33.914610 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21efaac3a3d643679c5abaaddb9e5a3a" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:33.914754 349 raft_consensus.cc:2454] T 21efaac3a3d643679c5abaaddb9e5a3a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:33.915213 32489 leader_election.cc:302] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:33.916112 923 raft_consensus.cc:2781] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:33.916174 923 raft_consensus.cc:482] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:33.916224 923 raft_consensus.cc:3037] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.917122 923 raft_consensus.cc:504] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:33.917459 923 leader_election.cc:288] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:33.917969 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21efaac3a3d643679c5abaaddb9e5a3a" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:33.918082 349 raft_consensus.cc:3037] T 21efaac3a3d643679c5abaaddb9e5a3a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.918924 349 raft_consensus.cc:2454] T 21efaac3a3d643679c5abaaddb9e5a3a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:33.919173 32489 leader_election.cc:302] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:33.919317 923 raft_consensus.cc:2781] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:33.919392 923 raft_consensus.cc:686] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:09:33.919525 923 consensus_queue.cc:237] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:33.919806 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21efaac3a3d643679c5abaaddb9e5a3a" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:33.919924 32685 raft_consensus.cc:2454] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:33.920131 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21efaac3a3d643679c5abaaddb9e5a3a" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:33.920217 32683 raft_consensus.cc:3037] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.921144 32683 raft_consensus.cc:2454] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:33.921633 29870 catalog_manager.cc:5576] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357) does not have the latest schema for tablet 76f2a93fab2d4c668365be3335d4b1ad (table rebalancer_dynamic_table [id=c4d4abe1d4e4474e8ee4f5b565b937a9]). Expected version 1 got 0
I20240501 14:09:33.922191 949 tablet.cc:1722] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:33.922468 29867 catalog_manager.cc:5484] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:33.922981 949 tablet.cc:1620] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7: MemRowSet was empty: no flush needed.
I20240501 14:09:33.930624 883 tablet_bootstrap.cc:492] T 940c8f54339f4a32be2942682a41c685 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:33.930848 883 tablet_bootstrap.cc:492] T 940c8f54339f4a32be2942682a41c685 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap complete.
I20240501 14:09:33.930995 883 ts_tablet_manager.cc:1367] T 940c8f54339f4a32be2942682a41c685 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.029s user 0.022s sys 0.000s
I20240501 14:09:33.931347 883 raft_consensus.cc:348] T 940c8f54339f4a32be2942682a41c685 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.931452 883 raft_consensus.cc:729] T 940c8f54339f4a32be2942682a41c685 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.931578 883 consensus_queue.cc:260] T 940c8f54339f4a32be2942682a41c685 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:33.931861 883 ts_tablet_manager.cc:1398] T 940c8f54339f4a32be2942682a41c685 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:33.932379 30428 tablet_copy_service.cc:342] P d1c217469ae4478ebd49e4ed9f7352d7: Request end of tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-940c8f54339f4a32be2942682a41c685 received from {username='slave'} at 127.22.252.136:38643
I20240501 14:09:33.932454 30428 tablet_copy_service.cc:433] P d1c217469ae4478ebd49e4ed9f7352d7: ending tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-940c8f54339f4a32be2942682a41c685 on tablet 940c8f54339f4a32be2942682a41c685 with peer 9ecd907cf5634ac98f3f54f3dbec6849
I20240501 14:09:33.944994 329 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:33.950407 30388 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:33.950497 32401 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:33.957265 30125 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:33.957827 32534 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:33.966264 30256 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:33.967298 29994 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:33.969619 617 consensus_queue.cc:1035] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:33.970142 32665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:33.986150 562 consensus_queue.cc:1035] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:34.012814 902 ts_tablet_manager.cc:902] T 3a04e958f3d049139ff74202611af725 P db935501e9104de5b15c8780681d6c38: Initiating tablet copy from peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:34.013254 902 tablet_copy_client.cc:308] T 3a04e958f3d049139ff74202611af725 P db935501e9104de5b15c8780681d6c38: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.132:34357
I20240501 14:09:34.013896 30428 tablet_copy_service.cc:140] P d1c217469ae4478ebd49e4ed9f7352d7: Received BeginTabletCopySession request for tablet 3a04e958f3d049139ff74202611af725 from peer db935501e9104de5b15c8780681d6c38 ({username='slave'} at 127.22.252.135:43131)
I20240501 14:09:34.013998 30428 tablet_copy_service.cc:161] P d1c217469ae4478ebd49e4ed9f7352d7: Beginning new tablet copy session on tablet 3a04e958f3d049139ff74202611af725 from peer db935501e9104de5b15c8780681d6c38 at {username='slave'} at 127.22.252.135:43131: session id = db935501e9104de5b15c8780681d6c38-3a04e958f3d049139ff74202611af725
I20240501 14:09:34.014915 30428 tablet_copy_source_session.cc:205] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:34.015893 902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a04e958f3d049139ff74202611af725. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.016880 29867 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:34.020433 902 tablet_copy_client.cc:783] T 3a04e958f3d049139ff74202611af725 P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:34.020578 902 tablet_copy_client.cc:647] T 3a04e958f3d049139ff74202611af725 P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:34.026175 946 tablet.cc:1722] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:34.026487 946 tablet.cc:1620] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:34.026649 947 tablet.cc:1722] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:34.026929 947 tablet.cc:1620] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:34.030011 902 tablet_copy_client.cc:523] T 3a04e958f3d049139ff74202611af725 P db935501e9104de5b15c8780681d6c38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:34.030568 949 tablet.cc:1722] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:34.030865 949 tablet.cc:1620] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7: MemRowSet was empty: no flush needed.
I20240501 14:09:34.031140 29867 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=c4d4abe1d4e4474e8ee4f5b565b937a9] alter complete (version 2)
I20240501 14:09:34.031283 902 tablet_bootstrap.cc:492] T 3a04e958f3d049139ff74202611af725 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:34.052040 902 tablet_bootstrap.cc:492] T 3a04e958f3d049139ff74202611af725 P db935501e9104de5b15c8780681d6c38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:34.052305 902 tablet_bootstrap.cc:492] T 3a04e958f3d049139ff74202611af725 P db935501e9104de5b15c8780681d6c38: Bootstrap complete.
I20240501 14:09:34.052464 902 ts_tablet_manager.cc:1367] T 3a04e958f3d049139ff74202611af725 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.021s user 0.017s sys 0.003s
I20240501 14:09:34.052788 902 raft_consensus.cc:348] T 3a04e958f3d049139ff74202611af725 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:34.052881 902 raft_consensus.cc:729] T 3a04e958f3d049139ff74202611af725 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.053020 902 consensus_queue.cc:260] T 3a04e958f3d049139ff74202611af725 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:34.053267 902 ts_tablet_manager.cc:1398] T 3a04e958f3d049139ff74202611af725 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:34.053759 30428 tablet_copy_service.cc:342] P d1c217469ae4478ebd49e4ed9f7352d7: Request end of tablet copy session db935501e9104de5b15c8780681d6c38-3a04e958f3d049139ff74202611af725 received from {username='slave'} at 127.22.252.135:43131
I20240501 14:09:34.053831 30428 tablet_copy_service.cc:433] P d1c217469ae4478ebd49e4ed9f7352d7: ending tablet copy session db935501e9104de5b15c8780681d6c38-3a04e958f3d049139ff74202611af725 on tablet 3a04e958f3d049139ff74202611af725 with peer db935501e9104de5b15c8780681d6c38
I20240501 14:09:34.077080 32683 raft_consensus.cc:1203] T dd2fd47ec6684cb28d2f4678d2e90e75 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.077514 30408 consensus_queue.cc:237] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:34.079116 30014 raft_consensus.cc:1261] T 354658866eda4a2da6a2871fe0a2769d P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:34.079294 30146 raft_consensus.cc:1261] T 354658866eda4a2da6a2871fe0a2769d P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:34.079671 728 consensus_queue.cc:1035] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:34.079911 731 consensus_queue.cc:1035] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:34.080085 30341 consensus_peers.cc:485] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 354658866eda4a2da6a2871fe0a2769d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:34.080405 32415 raft_consensus.cc:1203] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.082742 728 raft_consensus.cc:2932] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:34.083140 30146 raft_consensus.cc:2932] T 354658866eda4a2da6a2871fe0a2769d P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:34.084415 30014 raft_consensus.cc:2932] T 354658866eda4a2da6a2871fe0a2769d P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:34.085489 29873 catalog_manager.cc:5484] T 354658866eda4a2da6a2871fe0a2769d P d35c16dcb506426993811bb2519d7db3 reported cstate change: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:34.106601 30014 consensus_queue.cc:237] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:34.107781 30408 raft_consensus.cc:1261] T f4b8334e04994f5f8dc96eb91b944a47 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:34.107852 30146 raft_consensus.cc:1261] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:34.108160 1012 consensus_queue.cc:1035] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:34.108345 29951 consensus_peers.cc:485] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4b8334e04994f5f8dc96eb91b944a47. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:34.108343 32287 consensus_queue.cc:1035] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:34.110136 32287 raft_consensus.cc:2932] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:34.110548 30146 raft_consensus.cc:2932] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:34.112143 30408 raft_consensus.cc:2932] T f4b8334e04994f5f8dc96eb91b944a47 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:34.113655 29870 catalog_manager.cc:5484] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3 reported cstate change: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:34.131393 30277 consensus_queue.cc:237] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.132972 30146 raft_consensus.cc:1261] T 1698ca3ec26741bb9b3131fe67c9682b P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:34.133055 30212 consensus_peers.cc:485] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): Couldn't send request to peer a91029b90bf9415a89a2f70736759ad3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1698ca3ec26741bb9b3131fe67c9682b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:34.133301 620 consensus_queue.cc:1035] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:34.133699 30408 raft_consensus.cc:1261] T 1698ca3ec26741bb9b3131fe67c9682b P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:34.134225 29873 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:34.134476 620 consensus_queue.cc:1035] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:34.136926 620 raft_consensus.cc:2932] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:34.137529 30408 raft_consensus.cc:2932] T 1698ca3ec26741bb9b3131fe67c9682b P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:34.139693 30146 raft_consensus.cc:2932] T 1698ca3ec26741bb9b3131fe67c9682b P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:34.139895 946 tablet.cc:1722] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:34.140152 947 tablet.cc:1722] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:34.141178 949 tablet.cc:1722] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:34.142020 29873 catalog_manager.cc:5484] T 1698ca3ec26741bb9b3131fe67c9682b P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:34.148053 349 raft_consensus.cc:1203] T af39c304693645269a6af59190e95cad P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.148190 29870 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=c4d4abe1d4e4474e8ee4f5b565b937a9] alter complete (version 3)
I20240501 14:09:34.167061 30277 consensus_queue.cc:237] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:34.168466 29867 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 14:09:34.168509 30408 raft_consensus.cc:1261] T 3b6a526c2ed84e1780ec392ef39d09b0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:34.168574 29867 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 14:09:34.168905 30146 raft_consensus.cc:1261] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:34.169257 1034 consensus_queue.cc:1035] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:34.169703 32330 consensus_queue.cc:1035] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:34.171272 29867 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 8c5a380ad2624291b9770c3f1633c14d
I20240501 14:09:34.171545 29867 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 4 replicas of tablet 4f5314fbd59c4443974a9c304ba9b1a7
I20240501 14:09:34.171761 29867 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 4 replicas of tablet f4b8334e04994f5f8dc96eb91b944a47
I20240501 14:09:34.172374 30256 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f5314fbd59c4443974a9c304ba9b1a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:34.172510 544 tablet_replica.cc:330] T 4f5314fbd59c4443974a9c304ba9b1a7 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:34.172658 544 raft_consensus.cc:2229] T 4f5314fbd59c4443974a9c304ba9b1a7 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.172842 544 raft_consensus.cc:2258] T 4f5314fbd59c4443974a9c304ba9b1a7 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.172941 32401 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f5314fbd59c4443974a9c304ba9b1a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:34.173008 29994 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c5a380ad2624291b9770c3f1633c14d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:34.173250 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c5a380ad2624291b9770c3f1633c14d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:34.173499 29995 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f5314fbd59c4443974a9c304ba9b1a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:34.173549 545 tablet_replica.cc:330] T 8c5a380ad2624291b9770c3f1633c14d P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:34.173626 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet f4b8334e04994f5f8dc96eb91b944a47 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:34.173678 545 raft_consensus.cc:2229] T 8c5a380ad2624291b9770c3f1633c14d P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.173736 30124 tablet_service.cc:1514] Processing DeleteTablet for tablet f4b8334e04994f5f8dc96eb91b944a47 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:34.173799 545 raft_consensus.cc:2258] T 8c5a380ad2624291b9770c3f1633c14d P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.173844 544 ts_tablet_manager.cc:1875] T 4f5314fbd59c4443974a9c304ba9b1a7 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.173934 32534 tablet_service.cc:1514] Processing DeleteTablet for tablet f4b8334e04994f5f8dc96eb91b944a47 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:34.174294 545 ts_tablet_manager.cc:1875] T 8c5a380ad2624291b9770c3f1633c14d P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.174803 30387 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f5314fbd59c4443974a9c304ba9b1a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:34.174988 546 tablet_replica.cc:330] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:34.175145 546 raft_consensus.cc:2229] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:34.175427 546 raft_consensus.cc:2258] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.175746 32330 raft_consensus.cc:2932] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:34.176049 544 ts_tablet_manager.cc:1888] T 4f5314fbd59c4443974a9c304ba9b1a7 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.176112 544 log.cc:1196] T 4f5314fbd59c4443974a9c304ba9b1a7 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/4f5314fbd59c4443974a9c304ba9b1a7
I20240501 14:09:34.176321 543 tablet_replica.cc:330] T 8c5a380ad2624291b9770c3f1633c14d P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:34.176410 544 ts_tablet_manager.cc:1909] T 4f5314fbd59c4443974a9c304ba9b1a7 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting consensus metadata
W20240501 14:09:34.176438 29860 catalog_manager.cc:4797] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): delete failed for tablet f4b8334e04994f5f8dc96eb91b944a47 because the tablet was not found. No further retry: Not found: Tablet not found: f4b8334e04994f5f8dc96eb91b944a47
I20240501 14:09:34.176484 543 raft_consensus.cc:2229] T 8c5a380ad2624291b9770c3f1633c14d P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.176661 543 raft_consensus.cc:2258] T 8c5a380ad2624291b9770c3f1633c14d P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.177145 543 ts_tablet_manager.cc:1875] T 8c5a380ad2624291b9770c3f1633c14d P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.177290 1054 tablet_replica.cc:330] T 4f5314fbd59c4443974a9c304ba9b1a7 P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:34.177297 30146 raft_consensus.cc:2932] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:34.177407 1054 raft_consensus.cc:2229] T 4f5314fbd59c4443974a9c304ba9b1a7 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.177511 1054 raft_consensus.cc:2258] T 4f5314fbd59c4443974a9c304ba9b1a7 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.177691 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet 4f5314fbd59c4443974a9c304ba9b1a7 (table rebalance_test_table_6 [id=a49ff4b9b5fe43dea63424b7e0856b72]) successfully deleted
I20240501 14:09:34.178121 30408 raft_consensus.cc:2932] T 3b6a526c2ed84e1780ec392ef39d09b0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:34.178620 29867 catalog_manager.cc:5484] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:34.179833 30388 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c5a380ad2624291b9770c3f1633c14d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:34.179960 30386 tablet_service.cc:1514] Processing DeleteTablet for tablet f4b8334e04994f5f8dc96eb91b944a47 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:34.180033 1054 ts_tablet_manager.cc:1875] T 4f5314fbd59c4443974a9c304ba9b1a7 P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.180541 546 ts_tablet_manager.cc:1875] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.179276 543 ts_tablet_manager.cc:1888] T 8c5a380ad2624291b9770c3f1633c14d P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:34.180619 543 log.cc:1196] T 8c5a380ad2624291b9770c3f1633c14d P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/8c5a380ad2624291b9770c3f1633c14d
I20240501 14:09:34.180922 543 ts_tablet_manager.cc:1909] T 8c5a380ad2624291b9770c3f1633c14d P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:34.181493 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 8c5a380ad2624291b9770c3f1633c14d (table rebalance_test_table_6 [id=a49ff4b9b5fe43dea63424b7e0856b72]) successfully deleted
I20240501 14:09:34.182209 1054 ts_tablet_manager.cc:1888] T 4f5314fbd59c4443974a9c304ba9b1a7 P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.182272 1054 log.cc:1196] T 4f5314fbd59c4443974a9c304ba9b1a7 P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/4f5314fbd59c4443974a9c304ba9b1a7
I20240501 14:09:34.182519 1054 ts_tablet_manager.cc:1909] T 4f5314fbd59c4443974a9c304ba9b1a7 P a91029b90bf9415a89a2f70736759ad3: Deleting consensus metadata
I20240501 14:09:34.182703 546 ts_tablet_manager.cc:1888] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.182756 546 log.cc:1196] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/4f5314fbd59c4443974a9c304ba9b1a7
I20240501 14:09:34.182998 546 ts_tablet_manager.cc:1909] T 4f5314fbd59c4443974a9c304ba9b1a7 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:34.183459 546 tablet_replica.cc:330] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:34.183555 546 raft_consensus.cc:2229] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:34.183531 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 4f5314fbd59c4443974a9c304ba9b1a7 (table rebalance_test_table_6 [id=a49ff4b9b5fe43dea63424b7e0856b72]) successfully deleted
I20240501 14:09:34.183671 546 raft_consensus.cc:2258] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.184001 546 ts_tablet_manager.cc:1875] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.184319 543 tablet_replica.cc:330] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:34.184463 543 raft_consensus.cc:2229] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.184473 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 4f5314fbd59c4443974a9c304ba9b1a7 (table rebalance_test_table_6 [id=a49ff4b9b5fe43dea63424b7e0856b72]) successfully deleted
W20240501 14:09:34.186353 30212 consensus_peers.cc:485] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): Couldn't send request to peer 9ecd907cf5634ac98f3f54f3dbec6849. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b6a526c2ed84e1780ec392ef39d09b0. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:34.186954 545 ts_tablet_manager.cc:1888] T 8c5a380ad2624291b9770c3f1633c14d P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:34.187038 545 log.cc:1196] T 8c5a380ad2624291b9770c3f1633c14d P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/8c5a380ad2624291b9770c3f1633c14d
I20240501 14:09:34.187587 545 ts_tablet_manager.cc:1909] T 8c5a380ad2624291b9770c3f1633c14d P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:34.188300 545 tablet_replica.cc:330] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:34.188313 546 ts_tablet_manager.cc:1888] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:34.188369 546 log.cc:1196] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/8c5a380ad2624291b9770c3f1633c14d
I20240501 14:09:34.188511 545 raft_consensus.cc:2229] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.188724 546 ts_tablet_manager.cc:1909] T 8c5a380ad2624291b9770c3f1633c14d P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:34.188874 545 raft_consensus.cc:2258] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.188916 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 8c5a380ad2624291b9770c3f1633c14d (table rebalance_test_table_6 [id=a49ff4b9b5fe43dea63424b7e0856b72]) successfully deleted
I20240501 14:09:34.189529 545 ts_tablet_manager.cc:1875] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.191087 545 ts_tablet_manager.cc:1888] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.191133 545 log.cc:1196] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/4f5314fbd59c4443974a9c304ba9b1a7
I20240501 14:09:34.191308 545 ts_tablet_manager.cc:1909] T 4f5314fbd59c4443974a9c304ba9b1a7 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:34.191928 545 tablet_replica.cc:330] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:34.192050 545 raft_consensus.cc:2229] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:34.192373 543 raft_consensus.cc:2258] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.192402 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 4f5314fbd59c4443974a9c304ba9b1a7 (table rebalance_test_table_6 [id=a49ff4b9b5fe43dea63424b7e0856b72]) successfully deleted
I20240501 14:09:34.192703 545 raft_consensus.cc:2258] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.193197 546 tablet_replica.cc:330] T f4b8334e04994f5f8dc96eb91b944a47 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:34.193305 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 8c5a380ad2624291b9770c3f1633c14d (table rebalance_test_table_6 [id=a49ff4b9b5fe43dea63424b7e0856b72]) successfully deleted
I20240501 14:09:34.193372 545 ts_tablet_manager.cc:1875] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.193411 546 raft_consensus.cc:2229] T f4b8334e04994f5f8dc96eb91b944a47 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.193580 546 raft_consensus.cc:2258] T f4b8334e04994f5f8dc96eb91b944a47 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.194228 546 ts_tablet_manager.cc:1875] T f4b8334e04994f5f8dc96eb91b944a47 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.195642 545 ts_tablet_manager.cc:1888] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.195717 545 log.cc:1196] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/f4b8334e04994f5f8dc96eb91b944a47
I20240501 14:09:34.195971 545 ts_tablet_manager.cc:1909] T f4b8334e04994f5f8dc96eb91b944a47 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:34.196574 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet f4b8334e04994f5f8dc96eb91b944a47 (table rebalance_test_table_6 [id=a49ff4b9b5fe43dea63424b7e0856b72]) successfully deleted
I20240501 14:09:34.196746 546 ts_tablet_manager.cc:1888] T f4b8334e04994f5f8dc96eb91b944a47 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.196806 546 log.cc:1196] T f4b8334e04994f5f8dc96eb91b944a47 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/f4b8334e04994f5f8dc96eb91b944a47
I20240501 14:09:34.196951 543 ts_tablet_manager.cc:1875] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.197046 546 ts_tablet_manager.cc:1909] T f4b8334e04994f5f8dc96eb91b944a47 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:34.197623 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet f4b8334e04994f5f8dc96eb91b944a47 (table rebalance_test_table_6 [id=a49ff4b9b5fe43dea63424b7e0856b72]) successfully deleted
I20240501 14:09:34.198834 543 ts_tablet_manager.cc:1888] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.198892 543 log.cc:1196] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/f4b8334e04994f5f8dc96eb91b944a47
I20240501 14:09:34.199074 543 ts_tablet_manager.cc:1909] T f4b8334e04994f5f8dc96eb91b944a47 P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:34.199851 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet f4b8334e04994f5f8dc96eb91b944a47 (table rebalance_test_table_6 [id=a49ff4b9b5fe43dea63424b7e0856b72]) successfully deleted
I20240501 14:09:34.214140 30146 consensus_queue.cc:237] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:34.215288 30014 raft_consensus.cc:1261] T fb466c8cb5a74897a884ac6012cc3734 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:34.215328 30277 raft_consensus.cc:1261] T fb466c8cb5a74897a884ac6012cc3734 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:34.215682 1072 consensus_queue.cc:1035] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:34.215993 32174 consensus_queue.cc:1035] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:34.219031 32174 raft_consensus.cc:2932] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
W20240501 14:09:34.219213 30079 consensus_peers.cc:485] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb466c8cb5a74897a884ac6012cc3734. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:34.219411 30277 raft_consensus.cc:2932] T fb466c8cb5a74897a884ac6012cc3734 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:34.221860 29873 catalog_manager.cc:5484] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 reported cstate change: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:34.222613 30014 raft_consensus.cc:2932] T fb466c8cb5a74897a884ac6012cc3734 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:34.238966 32554 raft_consensus.cc:1203] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.239200 30146 consensus_queue.cc:237] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:34.240914 30408 raft_consensus.cc:1261] T 5518112449d9448e98f5c779f1fa3248 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:34.240957 30277 raft_consensus.cc:1261] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:34.241333 1076 consensus_queue.cc:1035] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:34.241588 30082 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5518112449d9448e98f5c779f1fa3248. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:34.241734 1076 consensus_queue.cc:1035] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:34.243297 500 raft_consensus.cc:2932] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:34.243656 30277 raft_consensus.cc:2932] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:34.245324 30408 raft_consensus.cc:2932] T 5518112449d9448e98f5c779f1fa3248 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:34.246490 29872 catalog_manager.cc:5484] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:34.251062 29873 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:34.251188 29873 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:34.253111 29873 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 76f2a93fab2d4c668365be3335d4b1ad
I20240501 14:09:34.253787 329 tablet_service.cc:1514] Processing DeleteTablet for tablet 76f2a93fab2d4c668365be3335d4b1ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:34.253849 30386 tablet_service.cc:1514] Processing DeleteTablet for tablet 76f2a93fab2d4c668365be3335d4b1ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:34.253847 32534 tablet_service.cc:1514] Processing DeleteTablet for tablet 76f2a93fab2d4c668365be3335d4b1ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:34.254025 546 tablet_replica.cc:330] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:34.254055 1055 tablet_replica.cc:330] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:34.254176 546 raft_consensus.cc:2229] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.254285 546 raft_consensus.cc:2258] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.254361 1055 raft_consensus.cc:2229] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:34.254698 1055 raft_consensus.cc:2258] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.254961 546 ts_tablet_manager.cc:1875] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.255376 1100 tablet_replica.cc:330] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:34.255542 1100 raft_consensus.cc:2229] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.255662 1100 raft_consensus.cc:2258] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.255679 1055 ts_tablet_manager.cc:1875] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.256285 1100 ts_tablet_manager.cc:1875] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.257249 546 ts_tablet_manager.cc:1888] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.257336 546 log.cc:1196] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/76f2a93fab2d4c668365be3335d4b1ad
I20240501 14:09:34.257499 1055 ts_tablet_manager.cc:1888] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.257557 1055 log.cc:1196] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/76f2a93fab2d4c668365be3335d4b1ad
I20240501 14:09:34.257611 546 ts_tablet_manager.cc:1909] T 76f2a93fab2d4c668365be3335d4b1ad P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:34.257757 1055 ts_tablet_manager.cc:1909] T 76f2a93fab2d4c668365be3335d4b1ad P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:34.258626 1100 ts_tablet_manager.cc:1888] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.258744 1100 log.cc:1196] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/76f2a93fab2d4c668365be3335d4b1ad
I20240501 14:09:34.258709 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet 76f2a93fab2d4c668365be3335d4b1ad (table rebalancer_dynamic_table_new_name [id=c4d4abe1d4e4474e8ee4f5b565b937a9]) successfully deleted
I20240501 14:09:34.258982 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 76f2a93fab2d4c668365be3335d4b1ad (table rebalancer_dynamic_table_new_name [id=c4d4abe1d4e4474e8ee4f5b565b937a9]) successfully deleted
I20240501 14:09:34.259112 1100 ts_tablet_manager.cc:1909] T 76f2a93fab2d4c668365be3335d4b1ad P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:09:34.259755 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 76f2a93fab2d4c668365be3335d4b1ad (table rebalancer_dynamic_table_new_name [id=c4d4abe1d4e4474e8ee4f5b565b937a9]) successfully deleted
I20240501 14:09:34.267616 30014 consensus_queue.cc:237] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.269071 30146 raft_consensus.cc:1261] T fb36e198bcf34ef68a9595d7cbe93e80 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:34.269436 970 consensus_queue.cc:1035] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:34.269613 30277 raft_consensus.cc:1261] T fb36e198bcf34ef68a9595d7cbe93e80 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:34.269999 32287 consensus_queue.cc:1035] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:34.272747 970 raft_consensus.cc:2932] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
W20240501 14:09:34.272920 29950 consensus_peers.cc:485] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e -> Peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): Couldn't send request to peer a91029b90bf9415a89a2f70736759ad3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb36e198bcf34ef68a9595d7cbe93e80. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:34.273231 30146 raft_consensus.cc:2932] T fb36e198bcf34ef68a9595d7cbe93e80 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:34.275055 30277 raft_consensus.cc:2932] T fb36e198bcf34ef68a9595d7cbe93e80 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:34.275521 29873 catalog_manager.cc:5484] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:34.294968 30146 consensus_queue.cc:237] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:34.296319 30277 raft_consensus.cc:1261] T 18918995fb03421b9dfb4d05369c1ce8 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:34.296519 30081 consensus_peers.cc:485] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 -> Peer 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): Couldn't send request to peer 9ecd907cf5634ac98f3f54f3dbec6849. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 18918995fb03421b9dfb4d05369c1ce8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:34.296634 30014 raft_consensus.cc:1261] T 18918995fb03421b9dfb4d05369c1ce8 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:34.296717 1076 consensus_queue.cc:1035] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:34.296988 500 consensus_queue.cc:1035] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:34.298625 500 raft_consensus.cc:2932] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:34.299052 30014 raft_consensus.cc:2932] T 18918995fb03421b9dfb4d05369c1ce8 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:34.301002 30277 raft_consensus.cc:2932] T 18918995fb03421b9dfb4d05369c1ce8 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:34.302052 29872 catalog_manager.cc:5484] T 18918995fb03421b9dfb4d05369c1ce8 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:34.304284 640 consensus_queue.cc:1035] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:34.308187 1138 consensus_queue.cc:1035] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:34.323706 30408 consensus_queue.cc:237] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:34.324983 30015 raft_consensus.cc:1261] T 205a0fda164d46a29a031070f1d0c369 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:34.324983 30146 raft_consensus.cc:1261] T 205a0fda164d46a29a031070f1d0c369 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:34.325114 30344 consensus_peers.cc:485] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 205a0fda164d46a29a031070f1d0c369. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:34.325325 731 consensus_queue.cc:1035] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:34.325493 728 consensus_queue.cc:1035] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:34.327297 731 raft_consensus.cc:2932] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:34.329068 30146 raft_consensus.cc:2932] T 205a0fda164d46a29a031070f1d0c369 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:34.329202 30015 raft_consensus.cc:2932] T 205a0fda164d46a29a031070f1d0c369 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:34.330057 32554 raft_consensus.cc:1203] T 5da2beba68ba414281025fcee91ca0d0 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.331816 29873 catalog_manager.cc:5484] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:34.349076 30146 consensus_queue.cc:237] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:34.350270 30277 raft_consensus.cc:1261] T fc28b2028a9246b99e8887f03180d0c3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:34.350394 30408 raft_consensus.cc:1261] T fc28b2028a9246b99e8887f03180d0c3 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:34.350607 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc28b2028a9246b99e8887f03180d0c3. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:34.350651 32174 consensus_queue.cc:1035] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:34.351188 32174 consensus_queue.cc:1035] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:34.352802 500 raft_consensus.cc:2932] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:34.354163 30408 raft_consensus.cc:2932] T fc28b2028a9246b99e8887f03180d0c3 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:34.355068 30277 raft_consensus.cc:2932] T fc28b2028a9246b99e8887f03180d0c3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:34.355319 29872 catalog_manager.cc:5484] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 reported cstate change: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:34.356316 29867 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:34.362790 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 4b391050e0f4480faf30d2749f618cae (DEFAULT_TABLE table=rebalancer_dynamic_table [id=49c7b42b5f88482581798812bf953db3]), partition=
I20240501 14:09:34.363039 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b391050e0f4480faf30d2749f618cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.363101 329 tablet_service.cc:1467] Processing CreateTablet for tablet 4b391050e0f4480faf30d2749f618cae (DEFAULT_TABLE table=rebalancer_dynamic_table [id=49c7b42b5f88482581798812bf953db3]), partition=
I20240501 14:09:34.363260 32401 tablet_service.cc:1467] Processing CreateTablet for tablet 4b391050e0f4480faf30d2749f618cae (DEFAULT_TABLE table=rebalancer_dynamic_table [id=49c7b42b5f88482581798812bf953db3]), partition=
I20240501 14:09:34.363355 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b391050e0f4480faf30d2749f618cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.363458 32401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b391050e0f4480faf30d2749f618cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.365483 1179 tablet_bootstrap.cc:492] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:34.365657 1180 tablet_bootstrap.cc:492] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:34.366484 1180 tablet_bootstrap.cc:654] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.366484 1179 tablet_bootstrap.cc:654] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.367079 1178 tablet_bootstrap.cc:492] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:34.368034 1178 tablet_bootstrap.cc:654] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.368522 1179 tablet_bootstrap.cc:492] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:34.368660 1179 ts_tablet_manager.cc:1367] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:34.369069 1178 tablet_bootstrap.cc:492] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:34.369217 1178 ts_tablet_manager.cc:1367] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:34.369231 1180 tablet_bootstrap.cc:492] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:34.369302 1180 ts_tablet_manager.cc:1367] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:34.369216 1179 raft_consensus.cc:348] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:34.369355 1179 raft_consensus.cc:374] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.369644 1179 raft_consensus.cc:729] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.369716 1178 raft_consensus.cc:348] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:34.369879 1178 raft_consensus.cc:374] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.369830 1180 raft_consensus.cc:348] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:34.369936 1180 raft_consensus.cc:374] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.369947 1178 raft_consensus.cc:729] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.370049 1180 raft_consensus.cc:729] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.370038 1179 consensus_queue.cc:260] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:34.370172 1178 consensus_queue.cc:260] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:34.370204 1180 consensus_queue.cc:260] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:34.370385 1179 ts_tablet_manager.cc:1398] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:34.371142 1178 ts_tablet_manager.cc:1398] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:34.372077 1180 ts_tablet_manager.cc:1398] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:34.382237 30408 consensus_queue.cc:237] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.383428 30146 raft_consensus.cc:1261] T 53c51f3845c94f47b043621d98f70229 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:34.383463 30277 raft_consensus.cc:1261] T 53c51f3845c94f47b043621d98f70229 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:34.383703 30343 consensus_peers.cc:485] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): Couldn't send request to peer a91029b90bf9415a89a2f70736759ad3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53c51f3845c94f47b043621d98f70229. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:34.383792 995 consensus_queue.cc:1035] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:34.383939 731 consensus_queue.cc:1035] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:34.385715 731 raft_consensus.cc:2932] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:34.386111 30146 raft_consensus.cc:2932] T 53c51f3845c94f47b043621d98f70229 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:34.387933 30277 raft_consensus.cc:2932] T 53c51f3845c94f47b043621d98f70229 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:34.388800 29867 catalog_manager.cc:5484] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:34.399264 349 raft_consensus.cc:1203] T 940c8f54339f4a32be2942682a41c685 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.426393 566 consensus_queue.cc:1035] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:34.430430 923 consensus_queue.cc:1035] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:34.453495 29867 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:34.455552 349 raft_consensus.cc:1203] T 8d87b2a4f6764e20afbfb43681f58467 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.462289 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca8c2fa60c543dab6f30acce1958573 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=ec60e0d43df9403baaddddba692166ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.462610 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca8c2fa60c543dab6f30acce1958573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.462637 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 7a6a29261186429cbc05b1819999f986 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=ec60e0d43df9403baaddddba692166ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.462881 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a6a29261186429cbc05b1819999f986. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.463011 329 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca8c2fa60c543dab6f30acce1958573 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=ec60e0d43df9403baaddddba692166ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.463269 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca8c2fa60c543dab6f30acce1958573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.465159 1178 tablet_bootstrap.cc:492] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:34.465741 1180 tablet_bootstrap.cc:492] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:34.466149 1178 tablet_bootstrap.cc:654] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.466532 1180 tablet_bootstrap.cc:654] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.467224 1178 tablet_bootstrap.cc:492] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:34.467208 30124 tablet_service.cc:1467] Processing CreateTablet for tablet 90a84c54afc241709d87597733f79df7 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=ec60e0d43df9403baaddddba692166ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.467300 1178 ts_tablet_manager.cc:1367] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:34.467466 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a84c54afc241709d87597733f79df7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.467738 1178 raft_consensus.cc:348] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.467859 1178 raft_consensus.cc:374] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.467892 1178 raft_consensus.cc:729] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.468003 1178 consensus_queue.cc:260] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.468310 1178 ts_tablet_manager.cc:1398] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:34.469707 32401 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca8c2fa60c543dab6f30acce1958573 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=ec60e0d43df9403baaddddba692166ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.469837 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 90a84c54afc241709d87597733f79df7 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=ec60e0d43df9403baaddddba692166ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.469982 32401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca8c2fa60c543dab6f30acce1958573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.470136 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a84c54afc241709d87597733f79df7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.470942 329 tablet_service.cc:1467] Processing CreateTablet for tablet 7a6a29261186429cbc05b1819999f986 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=ec60e0d43df9403baaddddba692166ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.471163 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a6a29261186429cbc05b1819999f986. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.471843 1245 tablet_bootstrap.cc:492] T 90a84c54afc241709d87597733f79df7 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:34.472004 32402 tablet_service.cc:1467] Processing CreateTablet for tablet 7a6a29261186429cbc05b1819999f986 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=ec60e0d43df9403baaddddba692166ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.472107 1179 tablet_bootstrap.cc:492] T 0ca8c2fa60c543dab6f30acce1958573 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:34.472210 32402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a6a29261186429cbc05b1819999f986. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.472961 1179 tablet_bootstrap.cc:654] T 0ca8c2fa60c543dab6f30acce1958573 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.473589 1244 tablet_bootstrap.cc:492] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:34.474224 1180 tablet_bootstrap.cc:492] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:34.474202 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 90a84c54afc241709d87597733f79df7 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=ec60e0d43df9403baaddddba692166ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.474321 1180 ts_tablet_manager.cc:1367] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:34.474433 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a84c54afc241709d87597733f79df7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.474507 1179 tablet_bootstrap.cc:492] T 0ca8c2fa60c543dab6f30acce1958573 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:34.474577 1179 ts_tablet_manager.cc:1367] T 0ca8c2fa60c543dab6f30acce1958573 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:34.474720 1180 raft_consensus.cc:348] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.474809 1180 raft_consensus.cc:374] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.474838 1180 raft_consensus.cc:729] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.474960 1180 consensus_queue.cc:260] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.474965 1179 raft_consensus.cc:348] T 0ca8c2fa60c543dab6f30acce1958573 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.475036 1245 tablet_bootstrap.cc:654] T 90a84c54afc241709d87597733f79df7 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.475046 1179 raft_consensus.cc:374] T 0ca8c2fa60c543dab6f30acce1958573 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.475082 1179 raft_consensus.cc:729] T 0ca8c2fa60c543dab6f30acce1958573 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.475234 1180 ts_tablet_manager.cc:1398] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:34.475200 1179 consensus_queue.cc:260] T 0ca8c2fa60c543dab6f30acce1958573 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.475380 1180 tablet_bootstrap.cc:492] T 7a6a29261186429cbc05b1819999f986 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:34.475455 1179 ts_tablet_manager.cc:1398] T 0ca8c2fa60c543dab6f30acce1958573 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:34.475581 1179 tablet_bootstrap.cc:492] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:34.476209 1180 tablet_bootstrap.cc:654] T 7a6a29261186429cbc05b1819999f986 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.478647 1179 tablet_bootstrap.cc:654] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.479300 1180 tablet_bootstrap.cc:492] T 7a6a29261186429cbc05b1819999f986 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:34.479424 1180 ts_tablet_manager.cc:1367] T 7a6a29261186429cbc05b1819999f986 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:34.479861 1180 raft_consensus.cc:348] T 7a6a29261186429cbc05b1819999f986 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:34.479982 1180 raft_consensus.cc:374] T 7a6a29261186429cbc05b1819999f986 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.480053 1180 raft_consensus.cc:729] T 7a6a29261186429cbc05b1819999f986 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.480196 1180 consensus_queue.cc:260] T 7a6a29261186429cbc05b1819999f986 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:34.480366 1179 tablet_bootstrap.cc:492] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:34.480422 1245 tablet_bootstrap.cc:492] T 90a84c54afc241709d87597733f79df7 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:34.480476 1179 ts_tablet_manager.cc:1367] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:09:34.480510 1245 ts_tablet_manager.cc:1367] T 90a84c54afc241709d87597733f79df7 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:34.480873 1179 raft_consensus.cc:348] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:34.480931 1180 ts_tablet_manager.cc:1398] T 7a6a29261186429cbc05b1819999f986 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:34.480964 1179 raft_consensus.cc:374] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.480995 1179 raft_consensus.cc:729] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.480944 1245 raft_consensus.cc:348] T 90a84c54afc241709d87597733f79df7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.481068 1245 raft_consensus.cc:374] T 90a84c54afc241709d87597733f79df7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.481101 1245 raft_consensus.cc:729] T 90a84c54afc241709d87597733f79df7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.481220 1245 consensus_queue.cc:260] T 90a84c54afc241709d87597733f79df7 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.481467 1245 ts_tablet_manager.cc:1398] T 90a84c54afc241709d87597733f79df7 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:34.481823 1179 consensus_queue.cc:260] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:34.482167 1179 ts_tablet_manager.cc:1398] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:34.482318 1179 tablet_bootstrap.cc:492] T 90a84c54afc241709d87597733f79df7 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:34.482890 1244 tablet_bootstrap.cc:654] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.483537 1179 tablet_bootstrap.cc:654] T 90a84c54afc241709d87597733f79df7 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.483548 1241 tablet_bootstrap.cc:492] T 7a6a29261186429cbc05b1819999f986 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:34.484720 1179 tablet_bootstrap.cc:492] T 90a84c54afc241709d87597733f79df7 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:34.484797 1179 ts_tablet_manager.cc:1367] T 90a84c54afc241709d87597733f79df7 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:34.484864 1244 tablet_bootstrap.cc:492] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:34.485208 1179 raft_consensus.cc:348] T 90a84c54afc241709d87597733f79df7 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.485291 1179 raft_consensus.cc:374] T 90a84c54afc241709d87597733f79df7 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.485319 1179 raft_consensus.cc:729] T 90a84c54afc241709d87597733f79df7 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.485417 1179 consensus_queue.cc:260] T 90a84c54afc241709d87597733f79df7 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.485639 1179 ts_tablet_manager.cc:1398] T 90a84c54afc241709d87597733f79df7 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:34.485752 1244 ts_tablet_manager.cc:1367] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:34.486152 1244 raft_consensus.cc:348] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.486250 1244 raft_consensus.cc:374] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.486282 1244 raft_consensus.cc:729] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.486420 1244 consensus_queue.cc:260] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.486696 1244 ts_tablet_manager.cc:1398] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:34.487287 1241 tablet_bootstrap.cc:654] T 7a6a29261186429cbc05b1819999f986 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.488521 1241 tablet_bootstrap.cc:492] T 7a6a29261186429cbc05b1819999f986 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:34.488613 1241 ts_tablet_manager.cc:1367] T 7a6a29261186429cbc05b1819999f986 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:34.489009 1241 raft_consensus.cc:348] T 7a6a29261186429cbc05b1819999f986 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:34.489096 1241 raft_consensus.cc:374] T 7a6a29261186429cbc05b1819999f986 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.489130 1241 raft_consensus.cc:729] T 7a6a29261186429cbc05b1819999f986 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.489251 1241 consensus_queue.cc:260] T 7a6a29261186429cbc05b1819999f986 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:34.489564 1241 ts_tablet_manager.cc:1398] T 7a6a29261186429cbc05b1819999f986 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:34.507814 500 raft_consensus.cc:482] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:34.507918 500 raft_consensus.cc:504] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.508299 500 leader_election.cc:288] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:34.508613 30015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a84c54afc241709d87597733f79df7" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:34.508646 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a84c54afc241709d87597733f79df7" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:34.508749 30015 raft_consensus.cc:2454] T 90a84c54afc241709d87597733f79df7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:34.508775 32415 raft_consensus.cc:2454] T 90a84c54afc241709d87597733f79df7 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:34.509104 30081 leader_election.cc:302] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:34.509268 500 raft_consensus.cc:2781] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:34.509327 500 raft_consensus.cc:482] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:34.509357 500 raft_consensus.cc:3037] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.510250 500 raft_consensus.cc:504] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.510298 1253 raft_consensus.cc:482] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:34.510385 1253 raft_consensus.cc:504] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:34.510578 500 leader_election.cc:288] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:34.510896 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a84c54afc241709d87597733f79df7" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:34.510906 30015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a84c54afc241709d87597733f79df7" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:34.510979 32415 raft_consensus.cc:3037] T 90a84c54afc241709d87597733f79df7 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.511026 30015 raft_consensus.cc:3037] T 90a84c54afc241709d87597733f79df7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.511296 1138 raft_consensus.cc:482] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:34.511384 1138 raft_consensus.cc:504] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:34.511746 1138 leader_election.cc:288] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:34.511952 32415 raft_consensus.cc:2454] T 90a84c54afc241709d87597733f79df7 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:34.512079 30015 raft_consensus.cc:2454] T 90a84c54afc241709d87597733f79df7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:34.512264 1253 leader_election.cc:288] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:34.512275 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b391050e0f4480faf30d2749f618cae" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:34.512317 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b391050e0f4480faf30d2749f618cae" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:34.512398 32683 raft_consensus.cc:2454] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:34.512421 30081 leader_election.cc:302] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:34.512563 500 raft_consensus.cc:2781] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:34.512640 500 raft_consensus.cc:686] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:34.512763 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b391050e0f4480faf30d2749f618cae" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:34.512852 32415 raft_consensus.cc:2454] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:34.512905 500 consensus_queue.cc:237] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.513020 32356 leader_election.cc:302] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:34.513074 32752 leader_election.cc:302] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters: db935501e9104de5b15c8780681d6c38
I20240501 14:09:34.513303 1138 raft_consensus.cc:2781] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:34.513356 1138 raft_consensus.cc:482] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:34.513154 1253 raft_consensus.cc:2781] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:34.513393 1138 raft_consensus.cc:3037] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.513422 1253 raft_consensus.cc:482] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:34.513463 1253 raft_consensus.cc:3037] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.514384 1138 raft_consensus.cc:504] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:34.514703 1138 leader_election.cc:288] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:34.514909 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b391050e0f4480faf30d2749f618cae" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:34.515311 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b391050e0f4480faf30d2749f618cae" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:34.515406 32685 raft_consensus.cc:3037] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.515707 1253 raft_consensus.cc:504] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:34.516039 1253 leader_election.cc:288] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:34.516173 29867 catalog_manager.cc:5484] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:34.516363 32685 raft_consensus.cc:2454] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:34.516537 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b391050e0f4480faf30d2749f618cae" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:34.516675 32683 raft_consensus.cc:2379] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a91029b90bf9415a89a2f70736759ad3 in current term 1: Already voted for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in this term.
I20240501 14:09:34.516744 32750 leader_election.cc:302] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:34.516884 1138 raft_consensus.cc:2781] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:34.516997 1138 raft_consensus.cc:686] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:34.517045 32415 raft_consensus.cc:2379] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9ecd907cf5634ac98f3f54f3dbec6849 in current term 1: Already voted for candidate a91029b90bf9415a89a2f70736759ad3 in this term.
I20240501 14:09:34.517131 1138 consensus_queue.cc:237] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:34.519201 29867 catalog_manager.cc:5484] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:34.519955 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b391050e0f4480faf30d2749f618cae" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:34.519989 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b391050e0f4480faf30d2749f618cae" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:34.520502 32358 leader_election.cc:302] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3; no voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38
I20240501 14:09:34.520742 1253 raft_consensus.cc:2726] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 14:09:34.535485 902 ts_tablet_manager.cc:902] T 354658866eda4a2da6a2871fe0a2769d P db935501e9104de5b15c8780681d6c38: Initiating tablet copy from peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:34.539188 902 tablet_copy_client.cc:308] T 354658866eda4a2da6a2871fe0a2769d P db935501e9104de5b15c8780681d6c38: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.132:34357
I20240501 14:09:34.539654 30428 tablet_copy_service.cc:140] P d1c217469ae4478ebd49e4ed9f7352d7: Received BeginTabletCopySession request for tablet 354658866eda4a2da6a2871fe0a2769d from peer db935501e9104de5b15c8780681d6c38 ({username='slave'} at 127.22.252.135:43131)
I20240501 14:09:34.539762 30428 tablet_copy_service.cc:161] P d1c217469ae4478ebd49e4ed9f7352d7: Beginning new tablet copy session on tablet 354658866eda4a2da6a2871fe0a2769d from peer db935501e9104de5b15c8780681d6c38 at {username='slave'} at 127.22.252.135:43131: session id = db935501e9104de5b15c8780681d6c38-354658866eda4a2da6a2871fe0a2769d
I20240501 14:09:34.540833 30428 tablet_copy_source_session.cc:205] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:34.541236 30256 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:34.544806 902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354658866eda4a2da6a2871fe0a2769d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.546809 902 tablet_copy_client.cc:783] T 354658866eda4a2da6a2871fe0a2769d P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:34.546947 902 tablet_copy_client.cc:647] T 354658866eda4a2da6a2871fe0a2769d P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:34.549151 30386 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:34.551330 32534 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:34.552399 29993 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:34.552587 30124 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:34.555682 32683 raft_consensus.cc:1203] T 3a04e958f3d049139ff74202611af725 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.556706 32665 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:34.565017 902 tablet_copy_client.cc:523] T 354658866eda4a2da6a2871fe0a2769d P db935501e9104de5b15c8780681d6c38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:34.565382 329 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:34.566382 902 tablet_bootstrap.cc:492] T 354658866eda4a2da6a2871fe0a2769d P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:34.573655 969 raft_consensus.cc:482] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:34.573748 969 raft_consensus.cc:504] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.574263 969 leader_election.cc:288] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:34.575140 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca8c2fa60c543dab6f30acce1958573" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:34.575335 32415 raft_consensus.cc:2454] T 0ca8c2fa60c543dab6f30acce1958573 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:34.575987 32620 leader_election.cc:302] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:34.576599 969 raft_consensus.cc:2781] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:34.576656 969 raft_consensus.cc:482] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:34.576691 969 raft_consensus.cc:3037] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.577613 969 raft_consensus.cc:504] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.577903 969 leader_election.cc:288] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Requested vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:34.578787 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca8c2fa60c543dab6f30acce1958573" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:34.578857 32400 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:34.578915 32415 raft_consensus.cc:3037] T 0ca8c2fa60c543dab6f30acce1958573 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.579890 32415 raft_consensus.cc:2454] T 0ca8c2fa60c543dab6f30acce1958573 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:34.580715 32620 leader_election.cc:302] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:34.580976 969 raft_consensus.cc:2781] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:34.581159 969 raft_consensus.cc:686] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Becoming Leader. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
I20240501 14:09:34.581386 969 consensus_queue.cc:237] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.585383 29872 catalog_manager.cc:5484] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db935501e9104de5b15c8780681d6c38 (127.22.252.135). New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:34.598978 1277 ts_tablet_manager.cc:902] T fb466c8cb5a74897a884ac6012cc3734 P db935501e9104de5b15c8780681d6c38: Initiating tablet copy from peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:34.599576 1277 tablet_copy_client.cc:308] T fb466c8cb5a74897a884ac6012cc3734 P db935501e9104de5b15c8780681d6c38: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.130:36457
I20240501 14:09:34.611483 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca8c2fa60c543dab6f30acce1958573" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:34.611625 349 raft_consensus.cc:2454] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:34.611788 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca8c2fa60c543dab6f30acce1958573" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:34.611858 348 raft_consensus.cc:3037] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.612883 348 raft_consensus.cc:2454] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:34.615069 30166 tablet_copy_service.cc:140] P d35c16dcb506426993811bb2519d7db3: Received BeginTabletCopySession request for tablet fb466c8cb5a74897a884ac6012cc3734 from peer db935501e9104de5b15c8780681d6c38 ({username='slave'} at 127.22.252.135:49787)
I20240501 14:09:34.615177 30166 tablet_copy_service.cc:161] P d35c16dcb506426993811bb2519d7db3: Beginning new tablet copy session on tablet fb466c8cb5a74897a884ac6012cc3734 from peer db935501e9104de5b15c8780681d6c38 at {username='slave'} at 127.22.252.135:49787: session id = db935501e9104de5b15c8780681d6c38-fb466c8cb5a74897a884ac6012cc3734
I20240501 14:09:34.616096 30166 tablet_copy_source_session.cc:205] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:34.617038 1277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb466c8cb5a74897a884ac6012cc3734. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.619148 1277 tablet_copy_client.cc:783] T fb466c8cb5a74897a884ac6012cc3734 P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:34.619287 1277 tablet_copy_client.cc:647] T fb466c8cb5a74897a884ac6012cc3734 P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:34.621112 1277 tablet_copy_client.cc:523] T fb466c8cb5a74897a884ac6012cc3734 P db935501e9104de5b15c8780681d6c38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:34.622231 1277 tablet_bootstrap.cc:492] T fb466c8cb5a74897a884ac6012cc3734 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:34.623785 29872 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:34.629995 32683 raft_consensus.cc:1261] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:34.630468 32416 raft_consensus.cc:1261] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:34.630522 640 consensus_queue.cc:1035] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:34.631521 640 consensus_queue.cc:1035] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:34.633733 29872 catalog_manager.cc:5576] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265) does not have the latest schema for tablet 4b391050e0f4480faf30d2749f618cae (table rebalancer_dynamic_table [id=49c7b42b5f88482581798812bf953db3]). Expected version 1 got 0
I20240501 14:09:34.635103 29867 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet 4b391050e0f4480faf30d2749f618cae (table rebalancer_dynamic_table [id=49c7b42b5f88482581798812bf953db3]). Expected version 1 got 0
I20240501 14:09:34.635306 947 tablet.cc:1722] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:34.635560 947 tablet.cc:1620] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:34.636983 902 tablet_bootstrap.cc:492] T 354658866eda4a2da6a2871fe0a2769d P db935501e9104de5b15c8780681d6c38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:34.637285 902 tablet_bootstrap.cc:492] T 354658866eda4a2da6a2871fe0a2769d P db935501e9104de5b15c8780681d6c38: Bootstrap complete.
I20240501 14:09:34.637465 902 ts_tablet_manager.cc:1367] T 354658866eda4a2da6a2871fe0a2769d P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.071s user 0.026s sys 0.000s
I20240501 14:09:34.637789 29867 catalog_manager.cc:5576] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265) does not have the latest schema for tablet 4b391050e0f4480faf30d2749f618cae (table rebalancer_dynamic_table [id=49c7b42b5f88482581798812bf953db3]). Expected version 1 got 0
I20240501 14:09:34.638062 1291 tablet.cc:1722] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:34.638103 902 raft_consensus.cc:348] T 354658866eda4a2da6a2871fe0a2769d P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:34.638232 902 raft_consensus.cc:729] T 354658866eda4a2da6a2871fe0a2769d P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.638361 1291 tablet.cc:1620] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:34.638414 902 consensus_queue.cc:260] T 354658866eda4a2da6a2871fe0a2769d P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:34.639214 902 ts_tablet_manager.cc:1398] T 354658866eda4a2da6a2871fe0a2769d P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:34.640367 30428 tablet_copy_service.cc:342] P d1c217469ae4478ebd49e4ed9f7352d7: Request end of tablet copy session db935501e9104de5b15c8780681d6c38-354658866eda4a2da6a2871fe0a2769d received from {username='slave'} at 127.22.252.135:43131
I20240501 14:09:34.640455 30428 tablet_copy_service.cc:433] P d1c217469ae4478ebd49e4ed9f7352d7: ending tablet copy session db935501e9104de5b15c8780681d6c38-354658866eda4a2da6a2871fe0a2769d on tablet 354658866eda4a2da6a2871fe0a2769d with peer db935501e9104de5b15c8780681d6c38
I20240501 14:09:34.641891 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=49c7b42b5f88482581798812bf953db3] alter complete (version 1)
I20240501 14:09:34.644987 1285 ts_tablet_manager.cc:902] T 1698ca3ec26741bb9b3131fe67c9682b P a91029b90bf9415a89a2f70736759ad3: Initiating tablet copy from peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:34.645588 1285 tablet_copy_client.cc:308] T 1698ca3ec26741bb9b3131fe67c9682b P a91029b90bf9415a89a2f70736759ad3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.131:43103
I20240501 14:09:34.647486 968 mvcc.cc:205] Tried to move back new op lower bound from 7022889265680224256 to 7022889265223405568. Current Snapshot: MvccSnapshot[applied={T|T < 7022889265680224256}]
I20240501 14:09:34.648869 29870 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet 4b391050e0f4480faf30d2749f618cae (table rebalancer_dynamic_table [id=49c7b42b5f88482581798812bf953db3]). Expected version 1 got 0
I20240501 14:09:34.649917 30297 tablet_copy_service.cc:140] P 1d0bb69611af4876a2b3b33c4766e13a: Received BeginTabletCopySession request for tablet 1698ca3ec26741bb9b3131fe67c9682b from peer a91029b90bf9415a89a2f70736759ad3 ({username='slave'} at 127.22.252.133:60853)
I20240501 14:09:34.650065 30297 tablet_copy_service.cc:161] P 1d0bb69611af4876a2b3b33c4766e13a: Beginning new tablet copy session on tablet 1698ca3ec26741bb9b3131fe67c9682b from peer a91029b90bf9415a89a2f70736759ad3 at {username='slave'} at 127.22.252.133:60853: session id = a91029b90bf9415a89a2f70736759ad3-1698ca3ec26741bb9b3131fe67c9682b
I20240501 14:09:34.650594 1297 tablet.cc:1722] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:34.651052 1297 tablet.cc:1620] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:34.651530 30297 tablet_copy_source_session.cc:205] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:34.652776 1285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1698ca3ec26741bb9b3131fe67c9682b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.656126 1285 tablet_copy_client.cc:783] T 1698ca3ec26741bb9b3131fe67c9682b P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:34.656391 1285 tablet_copy_client.cc:647] T 1698ca3ec26741bb9b3131fe67c9682b P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:34.657866 1285 tablet_copy_client.cc:523] T 1698ca3ec26741bb9b3131fe67c9682b P a91029b90bf9415a89a2f70736759ad3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:34.659632 1285 tablet_bootstrap.cc:492] T 1698ca3ec26741bb9b3131fe67c9682b P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:34.676354 29870 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 14:09:34.676477 1305 ts_tablet_manager.cc:902] T 18918995fb03421b9dfb4d05369c1ce8 P 9ecd907cf5634ac98f3f54f3dbec6849: Initiating tablet copy from peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:34.676504 29870 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 14:09:34.676713 1277 tablet_bootstrap.cc:492] T fb466c8cb5a74897a884ac6012cc3734 P db935501e9104de5b15c8780681d6c38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:34.677109 1277 tablet_bootstrap.cc:492] T fb466c8cb5a74897a884ac6012cc3734 P db935501e9104de5b15c8780681d6c38: Bootstrap complete.
I20240501 14:09:34.677239 1305 tablet_copy_client.cc:308] T 18918995fb03421b9dfb4d05369c1ce8 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.130:36457
I20240501 14:09:34.677345 1277 ts_tablet_manager.cc:1367] T fb466c8cb5a74897a884ac6012cc3734 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.055s user 0.024s sys 0.000s
I20240501 14:09:34.677723 30166 tablet_copy_service.cc:140] P d35c16dcb506426993811bb2519d7db3: Received BeginTabletCopySession request for tablet 18918995fb03421b9dfb4d05369c1ce8 from peer 9ecd907cf5634ac98f3f54f3dbec6849 ({username='slave'} at 127.22.252.136:49091)
I20240501 14:09:34.677822 30166 tablet_copy_service.cc:161] P d35c16dcb506426993811bb2519d7db3: Beginning new tablet copy session on tablet 18918995fb03421b9dfb4d05369c1ce8 from peer 9ecd907cf5634ac98f3f54f3dbec6849 at {username='slave'} at 127.22.252.136:49091: session id = 9ecd907cf5634ac98f3f54f3dbec6849-18918995fb03421b9dfb4d05369c1ce8
I20240501 14:09:34.677846 1277 raft_consensus.cc:348] T fb466c8cb5a74897a884ac6012cc3734 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:34.677968 1277 raft_consensus.cc:729] T fb466c8cb5a74897a884ac6012cc3734 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.678144 1277 consensus_queue.cc:260] T fb466c8cb5a74897a884ac6012cc3734 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:34.678491 1277 ts_tablet_manager.cc:1398] T fb466c8cb5a74897a884ac6012cc3734 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:34.678825 30166 tablet_copy_source_session.cc:205] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:34.678967 30165 tablet_copy_service.cc:342] P d35c16dcb506426993811bb2519d7db3: Request end of tablet copy session db935501e9104de5b15c8780681d6c38-fb466c8cb5a74897a884ac6012cc3734 received from {username='slave'} at 127.22.252.135:49787
I20240501 14:09:34.679021 30165 tablet_copy_service.cc:433] P d35c16dcb506426993811bb2519d7db3: ending tablet copy session db935501e9104de5b15c8780681d6c38-fb466c8cb5a74897a884ac6012cc3734 on tablet fb466c8cb5a74897a884ac6012cc3734 with peer db935501e9104de5b15c8780681d6c38
I20240501 14:09:34.679932 29870 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 4 replicas of tablet fb466c8cb5a74897a884ac6012cc3734
I20240501 14:09:34.679926 1305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18918995fb03421b9dfb4d05369c1ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.680312 29870 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 3c9856cd2c16497aa40bd041d5e0a297
I20240501 14:09:34.680505 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet fb466c8cb5a74897a884ac6012cc3734 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:34.680557 29870 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet ad69f22c52c34610bd47091f7305e19a
I20240501 14:09:34.680672 545 tablet_replica.cc:330] T fb466c8cb5a74897a884ac6012cc3734 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:34.680809 545 raft_consensus.cc:2229] T fb466c8cb5a74897a884ac6012cc3734 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.680835 30386 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c9856cd2c16497aa40bd041d5e0a297 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:34.680946 545 raft_consensus.cc:2258] T fb466c8cb5a74897a884ac6012cc3734 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.681016 546 tablet_replica.cc:330] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:34.681147 546 raft_consensus.cc:2229] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:34.681354 32665 tablet_service.cc:1514] Processing DeleteTablet for tablet fb466c8cb5a74897a884ac6012cc3734 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:34.681370 546 raft_consensus.cc:2258] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.681883 546 ts_tablet_manager.cc:1875] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.682122 29995 tablet_service.cc:1514] Processing DeleteTablet for tablet ad69f22c52c34610bd47091f7305e19a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:34.682255 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c9856cd2c16497aa40bd041d5e0a297 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:34.682624 30124 tablet_service.cc:1514] Processing DeleteTablet for tablet fb466c8cb5a74897a884ac6012cc3734 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:34.682768 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c9856cd2c16497aa40bd041d5e0a297 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:34.683109 545 ts_tablet_manager.cc:1875] T fb466c8cb5a74897a884ac6012cc3734 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.683982 546 ts_tablet_manager.cc:1888] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:34.684043 546 log.cc:1196] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/3c9856cd2c16497aa40bd041d5e0a297
I20240501 14:09:34.684340 30256 tablet_service.cc:1514] Processing DeleteTablet for tablet fb466c8cb5a74897a884ac6012cc3734 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:34.684607 30255 tablet_service.cc:1514] Processing DeleteTablet for tablet ad69f22c52c34610bd47091f7305e19a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:34.684645 1311 tablet_replica.cc:330] T fb466c8cb5a74897a884ac6012cc3734 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:34.684772 1311 raft_consensus.cc:2229] T fb466c8cb5a74897a884ac6012cc3734 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.684960 1311 raft_consensus.cc:2258] T fb466c8cb5a74897a884ac6012cc3734 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.685230 545 ts_tablet_manager.cc:1888] T fb466c8cb5a74897a884ac6012cc3734 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.685300 545 log.cc:1196] T fb466c8cb5a74897a884ac6012cc3734 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/fb466c8cb5a74897a884ac6012cc3734
I20240501 14:09:34.685636 543 tablet_replica.cc:330] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:34.685668 1311 ts_tablet_manager.cc:1875] T fb466c8cb5a74897a884ac6012cc3734 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.685725 30386 tablet_service.cc:1514] Processing DeleteTablet for tablet ad69f22c52c34610bd47091f7305e19a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:34.685781 543 raft_consensus.cc:2229] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:34.686025 543 raft_consensus.cc:2258] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.687263 543 ts_tablet_manager.cc:1875] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.687878 1311 ts_tablet_manager.cc:1888] T fb466c8cb5a74897a884ac6012cc3734 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.687945 1311 log.cc:1196] T fb466c8cb5a74897a884ac6012cc3734 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/fb466c8cb5a74897a884ac6012cc3734
I20240501 14:09:34.688197 1311 ts_tablet_manager.cc:1909] T fb466c8cb5a74897a884ac6012cc3734 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting consensus metadata
I20240501 14:09:34.688766 1311 tablet_replica.cc:330] T ad69f22c52c34610bd47091f7305e19a P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:34.688872 1311 raft_consensus.cc:2229] T ad69f22c52c34610bd47091f7305e19a P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.689054 1311 raft_consensus.cc:2258] T ad69f22c52c34610bd47091f7305e19a P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.689453 543 ts_tablet_manager.cc:1888] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.689556 1311 ts_tablet_manager.cc:1875] T ad69f22c52c34610bd47091f7305e19a P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.689884 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet fb466c8cb5a74897a884ac6012cc3734 (table rebalance_test_table_8 [id=40bc333ce8c9447da3fd22a75d02001d]) successfully deleted
I20240501 14:09:34.690016 1305 tablet_copy_client.cc:783] T 18918995fb03421b9dfb4d05369c1ce8 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:34.690160 1305 tablet_copy_client.cc:647] T 18918995fb03421b9dfb4d05369c1ce8 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:34.690346 546 ts_tablet_manager.cc:1909] T 3c9856cd2c16497aa40bd041d5e0a297 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:34.690662 1285 tablet_bootstrap.cc:492] T 1698ca3ec26741bb9b3131fe67c9682b P a91029b90bf9415a89a2f70736759ad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:34.690927 1285 tablet_bootstrap.cc:492] T 1698ca3ec26741bb9b3131fe67c9682b P a91029b90bf9415a89a2f70736759ad3: Bootstrap complete.
I20240501 14:09:34.691092 1285 ts_tablet_manager.cc:1367] T 1698ca3ec26741bb9b3131fe67c9682b P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.002s
I20240501 14:09:34.691092 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 3c9856cd2c16497aa40bd041d5e0a297 (table rebalance_test_table_8 [id=40bc333ce8c9447da3fd22a75d02001d]) successfully deleted
I20240501 14:09:34.691345 546 tablet_replica.cc:330] T ad69f22c52c34610bd47091f7305e19a P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:34.691411 545 ts_tablet_manager.cc:1909] T fb466c8cb5a74897a884ac6012cc3734 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:34.691505 546 raft_consensus.cc:2229] T ad69f22c52c34610bd47091f7305e19a P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.691552 1285 raft_consensus.cc:348] T 1698ca3ec26741bb9b3131fe67c9682b P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.691623 546 raft_consensus.cc:2258] T ad69f22c52c34610bd47091f7305e19a P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.691656 1285 raft_consensus.cc:729] T 1698ca3ec26741bb9b3131fe67c9682b P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.691659 1311 ts_tablet_manager.cc:1888] T ad69f22c52c34610bd47091f7305e19a P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:34.691717 1311 log.cc:1196] T ad69f22c52c34610bd47091f7305e19a P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/ad69f22c52c34610bd47091f7305e19a
I20240501 14:09:34.691813 1285 consensus_queue.cc:260] T 1698ca3ec26741bb9b3131fe67c9682b P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.691991 1311 ts_tablet_manager.cc:1909] T ad69f22c52c34610bd47091f7305e19a P 1d0bb69611af4876a2b3b33c4766e13a: Deleting consensus metadata
I20240501 14:09:34.692112 546 ts_tablet_manager.cc:1875] T ad69f22c52c34610bd47091f7305e19a P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.692179 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet fb466c8cb5a74897a884ac6012cc3734 (table rebalance_test_table_8 [id=40bc333ce8c9447da3fd22a75d02001d]) successfully deleted
I20240501 14:09:34.689515 543 log.cc:1196] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/fb466c8cb5a74897a884ac6012cc3734
I20240501 14:09:34.692569 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet ad69f22c52c34610bd47091f7305e19a (table rebalance_test_table_8 [id=40bc333ce8c9447da3fd22a75d02001d]) successfully deleted
I20240501 14:09:34.692687 1285 ts_tablet_manager.cc:1398] T 1698ca3ec26741bb9b3131fe67c9682b P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:34.692847 543 ts_tablet_manager.cc:1909] T fb466c8cb5a74897a884ac6012cc3734 P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:34.693305 543 tablet_replica.cc:330] T 3c9856cd2c16497aa40bd041d5e0a297 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:34.693413 543 raft_consensus.cc:2229] T 3c9856cd2c16497aa40bd041d5e0a297 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.693511 543 raft_consensus.cc:2258] T 3c9856cd2c16497aa40bd041d5e0a297 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.693529 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet fb466c8cb5a74897a884ac6012cc3734 (table rebalance_test_table_8 [id=40bc333ce8c9447da3fd22a75d02001d]) successfully deleted
I20240501 14:09:34.693931 543 ts_tablet_manager.cc:1875] T 3c9856cd2c16497aa40bd041d5e0a297 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.694278 545 tablet_replica.cc:330] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:34.694455 545 raft_consensus.cc:2229] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:34.694705 545 raft_consensus.cc:2258] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.694844 546 ts_tablet_manager.cc:1888] T ad69f22c52c34610bd47091f7305e19a P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:34.694835 30297 tablet_copy_service.cc:342] P 1d0bb69611af4876a2b3b33c4766e13a: Request end of tablet copy session a91029b90bf9415a89a2f70736759ad3-1698ca3ec26741bb9b3131fe67c9682b received from {username='slave'} at 127.22.252.133:60853
I20240501 14:09:34.694916 546 log.cc:1196] T ad69f22c52c34610bd47091f7305e19a P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/ad69f22c52c34610bd47091f7305e19a
I20240501 14:09:34.694916 30297 tablet_copy_service.cc:433] P 1d0bb69611af4876a2b3b33c4766e13a: ending tablet copy session a91029b90bf9415a89a2f70736759ad3-1698ca3ec26741bb9b3131fe67c9682b on tablet 1698ca3ec26741bb9b3131fe67c9682b with peer a91029b90bf9415a89a2f70736759ad3
I20240501 14:09:34.695184 546 ts_tablet_manager.cc:1909] T ad69f22c52c34610bd47091f7305e19a P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:34.695228 545 ts_tablet_manager.cc:1875] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.695832 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet ad69f22c52c34610bd47091f7305e19a (table rebalance_test_table_8 [id=40bc333ce8c9447da3fd22a75d02001d]) successfully deleted
I20240501 14:09:34.696825 1317 ts_tablet_manager.cc:902] T 3b6a526c2ed84e1780ec392ef39d09b0 P 9ecd907cf5634ac98f3f54f3dbec6849: Initiating tablet copy from peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:34.696857 1305 tablet_copy_client.cc:523] T 18918995fb03421b9dfb4d05369c1ce8 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:34.697340 1317 tablet_copy_client.cc:308] T 3b6a526c2ed84e1780ec392ef39d09b0 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.131:43103
I20240501 14:09:34.697813 543 ts_tablet_manager.cc:1888] T 3c9856cd2c16497aa40bd041d5e0a297 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:34.697870 543 log.cc:1196] T 3c9856cd2c16497aa40bd041d5e0a297 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/3c9856cd2c16497aa40bd041d5e0a297
I20240501 14:09:34.698123 543 ts_tablet_manager.cc:1909] T 3c9856cd2c16497aa40bd041d5e0a297 P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:34.698165 1305 tablet_bootstrap.cc:492] T 18918995fb03421b9dfb4d05369c1ce8 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:34.699877 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 3c9856cd2c16497aa40bd041d5e0a297 (table rebalance_test_table_8 [id=40bc333ce8c9447da3fd22a75d02001d]) successfully deleted
I20240501 14:09:34.700136 545 ts_tablet_manager.cc:1888] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:34.700212 545 log.cc:1196] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/ad69f22c52c34610bd47091f7305e19a
I20240501 14:09:34.700663 545 ts_tablet_manager.cc:1909] T ad69f22c52c34610bd47091f7305e19a P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:34.701375 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet ad69f22c52c34610bd47091f7305e19a (table rebalance_test_table_8 [id=40bc333ce8c9447da3fd22a75d02001d]) successfully deleted
I20240501 14:09:34.701614 545 tablet_replica.cc:330] T 3c9856cd2c16497aa40bd041d5e0a297 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:34.701771 545 raft_consensus.cc:2229] T 3c9856cd2c16497aa40bd041d5e0a297 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.701983 545 raft_consensus.cc:2258] T 3c9856cd2c16497aa40bd041d5e0a297 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.702473 545 ts_tablet_manager.cc:1875] T 3c9856cd2c16497aa40bd041d5e0a297 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.703598 30297 tablet_copy_service.cc:140] P 1d0bb69611af4876a2b3b33c4766e13a: Received BeginTabletCopySession request for tablet 3b6a526c2ed84e1780ec392ef39d09b0 from peer 9ecd907cf5634ac98f3f54f3dbec6849 ({username='slave'} at 127.22.252.136:55245)
I20240501 14:09:34.703696 30297 tablet_copy_service.cc:161] P 1d0bb69611af4876a2b3b33c4766e13a: Beginning new tablet copy session on tablet 3b6a526c2ed84e1780ec392ef39d09b0 from peer 9ecd907cf5634ac98f3f54f3dbec6849 at {username='slave'} at 127.22.252.136:55245: session id = 9ecd907cf5634ac98f3f54f3dbec6849-3b6a526c2ed84e1780ec392ef39d09b0
I20240501 14:09:34.704783 30297 tablet_copy_source_session.cc:205] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:34.704967 545 ts_tablet_manager.cc:1888] T 3c9856cd2c16497aa40bd041d5e0a297 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:34.705241 545 log.cc:1196] T 3c9856cd2c16497aa40bd041d5e0a297 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/3c9856cd2c16497aa40bd041d5e0a297
I20240501 14:09:34.705499 545 ts_tablet_manager.cc:1909] T 3c9856cd2c16497aa40bd041d5e0a297 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:34.706069 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 3c9856cd2c16497aa40bd041d5e0a297 (table rebalance_test_table_8 [id=40bc333ce8c9447da3fd22a75d02001d]) successfully deleted
I20240501 14:09:34.706157 1317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b6a526c2ed84e1780ec392ef39d09b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.706418 1316 ts_tablet_manager.cc:902] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3: Initiating tablet copy from peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:34.706894 1316 tablet_copy_client.cc:308] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.130:36457
I20240501 14:09:34.707151 1310 tablet_replica.cc:330] T fb466c8cb5a74897a884ac6012cc3734 P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:34.707315 1310 raft_consensus.cc:2229] T fb466c8cb5a74897a884ac6012cc3734 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.707352 30165 tablet_copy_service.cc:140] P d35c16dcb506426993811bb2519d7db3: Received BeginTabletCopySession request for tablet 5518112449d9448e98f5c779f1fa3248 from peer 5f8a0cfdfdca4148a75686dc88059cf3 ({username='slave'} at 127.22.252.134:40399)
I20240501 14:09:34.707442 30165 tablet_copy_service.cc:161] P d35c16dcb506426993811bb2519d7db3: Beginning new tablet copy session on tablet 5518112449d9448e98f5c779f1fa3248 from peer 5f8a0cfdfdca4148a75686dc88059cf3 at {username='slave'} at 127.22.252.134:40399: session id = 5f8a0cfdfdca4148a75686dc88059cf3-5518112449d9448e98f5c779f1fa3248
I20240501 14:09:34.707563 1310 raft_consensus.cc:2258] T fb466c8cb5a74897a884ac6012cc3734 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.707993 1317 tablet_copy_client.cc:783] T 3b6a526c2ed84e1780ec392ef39d09b0 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:34.708125 1317 tablet_copy_client.cc:647] T 3b6a526c2ed84e1780ec392ef39d09b0 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:34.708285 1310 ts_tablet_manager.cc:1875] T fb466c8cb5a74897a884ac6012cc3734 P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.708387 30165 tablet_copy_source_session.cc:205] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:34.709772 1316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5518112449d9448e98f5c779f1fa3248. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.710556 1310 ts_tablet_manager.cc:1888] T fb466c8cb5a74897a884ac6012cc3734 P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.710675 1310 log.cc:1196] T fb466c8cb5a74897a884ac6012cc3734 P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/fb466c8cb5a74897a884ac6012cc3734
I20240501 14:09:34.710997 1310 ts_tablet_manager.cc:1909] T fb466c8cb5a74897a884ac6012cc3734 P db935501e9104de5b15c8780681d6c38: Deleting consensus metadata
I20240501 14:09:34.711211 1317 tablet_copy_client.cc:523] T 3b6a526c2ed84e1780ec392ef39d09b0 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:34.711643 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet fb466c8cb5a74897a884ac6012cc3734 (table rebalance_test_table_8 [id=40bc333ce8c9447da3fd22a75d02001d]) successfully deleted
I20240501 14:09:34.712287 1316 tablet_copy_client.cc:783] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:34.712414 1316 tablet_copy_client.cc:647] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:34.712452 1317 tablet_bootstrap.cc:492] T 3b6a526c2ed84e1780ec392ef39d09b0 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:34.717010 1316 tablet_copy_client.cc:523] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:34.719177 1316 tablet_bootstrap.cc:492] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:34.731091 1305 tablet_bootstrap.cc:492] T 18918995fb03421b9dfb4d05369c1ce8 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:34.731356 1305 tablet_bootstrap.cc:492] T 18918995fb03421b9dfb4d05369c1ce8 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap complete.
I20240501 14:09:34.731515 1305 ts_tablet_manager.cc:1367] T 18918995fb03421b9dfb4d05369c1ce8 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.002s
I20240501 14:09:34.731891 1305 raft_consensus.cc:348] T 18918995fb03421b9dfb4d05369c1ce8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:34.732012 1305 raft_consensus.cc:729] T 18918995fb03421b9dfb4d05369c1ce8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.732172 1305 consensus_queue.cc:260] T 18918995fb03421b9dfb4d05369c1ce8 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:34.733016 1305 ts_tablet_manager.cc:1398] T 18918995fb03421b9dfb4d05369c1ce8 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 14:09:34.733546 30165 tablet_copy_service.cc:342] P d35c16dcb506426993811bb2519d7db3: Request end of tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-18918995fb03421b9dfb4d05369c1ce8 received from {username='slave'} at 127.22.252.136:49091
I20240501 14:09:34.733631 30165 tablet_copy_service.cc:433] P d35c16dcb506426993811bb2519d7db3: ending tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-18918995fb03421b9dfb4d05369c1ce8 on tablet 18918995fb03421b9dfb4d05369c1ce8 with peer 9ecd907cf5634ac98f3f54f3dbec6849
I20240501 14:09:34.738147 1285 ts_tablet_manager.cc:902] T fb36e198bcf34ef68a9595d7cbe93e80 P a91029b90bf9415a89a2f70736759ad3: Initiating tablet copy from peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:34.738597 1285 tablet_copy_client.cc:308] T fb36e198bcf34ef68a9595d7cbe93e80 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.129:41387
I20240501 14:09:34.743252 30035 tablet_copy_service.cc:140] P 93f2518fbadf4f9991cda93afacb363e: Received BeginTabletCopySession request for tablet fb36e198bcf34ef68a9595d7cbe93e80 from peer a91029b90bf9415a89a2f70736759ad3 ({username='slave'} at 127.22.252.133:60917)
I20240501 14:09:34.743357 30035 tablet_copy_service.cc:161] P 93f2518fbadf4f9991cda93afacb363e: Beginning new tablet copy session on tablet fb36e198bcf34ef68a9595d7cbe93e80 from peer a91029b90bf9415a89a2f70736759ad3 at {username='slave'} at 127.22.252.133:60917: session id = a91029b90bf9415a89a2f70736759ad3-fb36e198bcf34ef68a9595d7cbe93e80
I20240501 14:09:34.744407 30035 tablet_copy_source_session.cc:205] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:34.745294 1285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb36e198bcf34ef68a9595d7cbe93e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.745767 29867 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:34.747503 1285 tablet_copy_client.cc:783] T fb36e198bcf34ef68a9595d7cbe93e80 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:34.747644 1285 tablet_copy_client.cc:647] T fb36e198bcf34ef68a9595d7cbe93e80 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:34.748049 1316 tablet_bootstrap.cc:492] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:34.748412 1316 tablet_bootstrap.cc:492] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap complete.
I20240501 14:09:34.748620 1316 ts_tablet_manager.cc:1367] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.029s user 0.020s sys 0.007s
I20240501 14:09:34.749104 1316 raft_consensus.cc:348] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:34.749233 1316 raft_consensus.cc:729] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.749406 1316 consensus_queue.cc:260] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:34.749682 1316 ts_tablet_manager.cc:1398] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:34.752262 947 tablet.cc:1722] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:34.752379 30165 tablet_copy_service.cc:342] P d35c16dcb506426993811bb2519d7db3: Request end of tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-5518112449d9448e98f5c779f1fa3248 received from {username='slave'} at 127.22.252.134:40399
I20240501 14:09:34.752452 30165 tablet_copy_service.cc:433] P d35c16dcb506426993811bb2519d7db3: ending tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-5518112449d9448e98f5c779f1fa3248 on tablet 5518112449d9448e98f5c779f1fa3248 with peer 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:09:34.752517 947 tablet.cc:1620] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:34.753127 1291 tablet.cc:1722] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:34.753399 1291 tablet.cc:1620] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:34.754035 1297 tablet.cc:1722] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:34.754271 1297 tablet.cc:1620] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:34.758893 29870 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=49c7b42b5f88482581798812bf953db3] alter complete (version 2)
I20240501 14:09:34.759421 1285 tablet_copy_client.cc:523] T fb36e198bcf34ef68a9595d7cbe93e80 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:34.761312 1285 tablet_bootstrap.cc:492] T fb36e198bcf34ef68a9595d7cbe93e80 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:34.765836 1277 ts_tablet_manager.cc:902] T fc28b2028a9246b99e8887f03180d0c3 P db935501e9104de5b15c8780681d6c38: Initiating tablet copy from peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:34.766260 1277 tablet_copy_client.cc:308] T fc28b2028a9246b99e8887f03180d0c3 P db935501e9104de5b15c8780681d6c38: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.130:36457
I20240501 14:09:34.766640 30165 tablet_copy_service.cc:140] P d35c16dcb506426993811bb2519d7db3: Received BeginTabletCopySession request for tablet fc28b2028a9246b99e8887f03180d0c3 from peer db935501e9104de5b15c8780681d6c38 ({username='slave'} at 127.22.252.135:49787)
I20240501 14:09:34.766739 30165 tablet_copy_service.cc:161] P d35c16dcb506426993811bb2519d7db3: Beginning new tablet copy session on tablet fc28b2028a9246b99e8887f03180d0c3 from peer db935501e9104de5b15c8780681d6c38 at {username='slave'} at 127.22.252.135:49787: session id = db935501e9104de5b15c8780681d6c38-fc28b2028a9246b99e8887f03180d0c3
I20240501 14:09:34.767648 30165 tablet_copy_source_session.cc:205] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:34.768512 1277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc28b2028a9246b99e8887f03180d0c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.769516 1317 tablet_bootstrap.cc:492] T 3b6a526c2ed84e1780ec392ef39d09b0 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:34.769776 1317 tablet_bootstrap.cc:492] T 3b6a526c2ed84e1780ec392ef39d09b0 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap complete.
I20240501 14:09:34.769924 1317 ts_tablet_manager.cc:1367] T 3b6a526c2ed84e1780ec392ef39d09b0 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.058s user 0.024s sys 0.000s
I20240501 14:09:34.770522 1277 tablet_copy_client.cc:783] T fc28b2028a9246b99e8887f03180d0c3 P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:34.770658 1277 tablet_copy_client.cc:647] T fc28b2028a9246b99e8887f03180d0c3 P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:34.771345 1317 raft_consensus.cc:348] T 3b6a526c2ed84e1780ec392ef39d09b0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:34.771576 1317 raft_consensus.cc:729] T 3b6a526c2ed84e1780ec392ef39d09b0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.771977 1317 consensus_queue.cc:260] T 3b6a526c2ed84e1780ec392ef39d09b0 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:34.772328 1317 ts_tablet_manager.cc:1398] T 3b6a526c2ed84e1780ec392ef39d09b0 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:34.773553 30297 tablet_copy_service.cc:342] P 1d0bb69611af4876a2b3b33c4766e13a: Request end of tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-3b6a526c2ed84e1780ec392ef39d09b0 received from {username='slave'} at 127.22.252.136:55245
I20240501 14:09:34.773627 30297 tablet_copy_service.cc:433] P 1d0bb69611af4876a2b3b33c4766e13a: ending tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-3b6a526c2ed84e1780ec392ef39d09b0 on tablet 3b6a526c2ed84e1780ec392ef39d09b0 with peer 9ecd907cf5634ac98f3f54f3dbec6849
I20240501 14:09:34.775087 1277 tablet_copy_client.cc:523] T fc28b2028a9246b99e8887f03180d0c3 P db935501e9104de5b15c8780681d6c38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:34.776228 1277 tablet_bootstrap.cc:492] T fc28b2028a9246b99e8887f03180d0c3 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:34.802639 1277 tablet_bootstrap.cc:492] T fc28b2028a9246b99e8887f03180d0c3 P db935501e9104de5b15c8780681d6c38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:34.802919 1277 tablet_bootstrap.cc:492] T fc28b2028a9246b99e8887f03180d0c3 P db935501e9104de5b15c8780681d6c38: Bootstrap complete.
I20240501 14:09:34.803081 1277 ts_tablet_manager.cc:1367] T fc28b2028a9246b99e8887f03180d0c3 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.027s user 0.024s sys 0.000s
I20240501 14:09:34.803474 1277 raft_consensus.cc:348] T fc28b2028a9246b99e8887f03180d0c3 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:34.803586 1277 raft_consensus.cc:729] T fc28b2028a9246b99e8887f03180d0c3 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.803728 1277 consensus_queue.cc:260] T fc28b2028a9246b99e8887f03180d0c3 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:34.804634 1277 ts_tablet_manager.cc:1398] T fc28b2028a9246b99e8887f03180d0c3 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 14:09:34.805120 30165 tablet_copy_service.cc:342] P d35c16dcb506426993811bb2519d7db3: Request end of tablet copy session db935501e9104de5b15c8780681d6c38-fc28b2028a9246b99e8887f03180d0c3 received from {username='slave'} at 127.22.252.135:49787
I20240501 14:09:34.805228 30165 tablet_copy_service.cc:433] P d35c16dcb506426993811bb2519d7db3: ending tablet copy session db935501e9104de5b15c8780681d6c38-fc28b2028a9246b99e8887f03180d0c3 on tablet fc28b2028a9246b99e8887f03180d0c3 with peer db935501e9104de5b15c8780681d6c38
I20240501 14:09:34.810079 1253 raft_consensus.cc:482] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:34.810178 1253 raft_consensus.cc:504] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:34.810547 1253 leader_election.cc:288] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:34.811058 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a6a29261186429cbc05b1819999f986" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:34.811132 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a6a29261186429cbc05b1819999f986" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:34.811187 348 raft_consensus.cc:2454] T 7a6a29261186429cbc05b1819999f986 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:34.811254 32553 raft_consensus.cc:2454] T 7a6a29261186429cbc05b1819999f986 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:34.811565 32358 leader_election.cc:302] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:34.811775 1253 raft_consensus.cc:2781] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:34.811836 1253 raft_consensus.cc:482] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:34.811870 1253 raft_consensus.cc:3037] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.812834 1253 raft_consensus.cc:504] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:34.812911 1285 tablet_bootstrap.cc:492] T fb36e198bcf34ef68a9595d7cbe93e80 P a91029b90bf9415a89a2f70736759ad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:34.813187 1285 tablet_bootstrap.cc:492] T fb36e198bcf34ef68a9595d7cbe93e80 P a91029b90bf9415a89a2f70736759ad3: Bootstrap complete.
I20240501 14:09:34.813206 1253 leader_election.cc:288] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:34.813344 1285 ts_tablet_manager.cc:1367] T fb36e198bcf34ef68a9595d7cbe93e80 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.052s user 0.025s sys 0.000s
I20240501 14:09:34.813802 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a6a29261186429cbc05b1819999f986" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:34.813792 1285 raft_consensus.cc:348] T fb36e198bcf34ef68a9595d7cbe93e80 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.813880 1285 raft_consensus.cc:729] T fb36e198bcf34ef68a9595d7cbe93e80 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.813902 32553 raft_consensus.cc:3037] T 7a6a29261186429cbc05b1819999f986 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.813989 1285 consensus_queue.cc:260] T fb36e198bcf34ef68a9595d7cbe93e80 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.814260 1285 ts_tablet_manager.cc:1398] T fb36e198bcf34ef68a9595d7cbe93e80 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:34.814796 32553 raft_consensus.cc:2454] T 7a6a29261186429cbc05b1819999f986 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:34.815093 32359 leader_election.cc:302] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:34.815305 562 raft_consensus.cc:2781] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:34.815387 562 raft_consensus.cc:686] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:34.815490 30035 tablet_copy_service.cc:342] P 93f2518fbadf4f9991cda93afacb363e: Request end of tablet copy session a91029b90bf9415a89a2f70736759ad3-fb36e198bcf34ef68a9595d7cbe93e80 received from {username='slave'} at 127.22.252.133:60917
I20240501 14:09:34.815503 562 consensus_queue.cc:237] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:34.815557 30035 tablet_copy_service.cc:433] P 93f2518fbadf4f9991cda93afacb363e: ending tablet copy session a91029b90bf9415a89a2f70736759ad3-fb36e198bcf34ef68a9595d7cbe93e80 on tablet fb36e198bcf34ef68a9595d7cbe93e80 with peer a91029b90bf9415a89a2f70736759ad3
I20240501 14:09:34.815901 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a6a29261186429cbc05b1819999f986" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:34.816009 348 raft_consensus.cc:3037] T 7a6a29261186429cbc05b1819999f986 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.816974 348 raft_consensus.cc:2454] T 7a6a29261186429cbc05b1819999f986 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:34.819021 29872 catalog_manager.cc:5484] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:34.831908 1316 ts_tablet_manager.cc:902] T 205a0fda164d46a29a031070f1d0c369 P 5f8a0cfdfdca4148a75686dc88059cf3: Initiating tablet copy from peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:34.832381 1316 tablet_copy_client.cc:308] T 205a0fda164d46a29a031070f1d0c369 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.132:34357
I20240501 14:09:34.832875 30428 tablet_copy_service.cc:140] P d1c217469ae4478ebd49e4ed9f7352d7: Received BeginTabletCopySession request for tablet 205a0fda164d46a29a031070f1d0c369 from peer 5f8a0cfdfdca4148a75686dc88059cf3 ({username='slave'} at 127.22.252.134:53955)
I20240501 14:09:34.832988 30428 tablet_copy_service.cc:161] P d1c217469ae4478ebd49e4ed9f7352d7: Beginning new tablet copy session on tablet 205a0fda164d46a29a031070f1d0c369 from peer 5f8a0cfdfdca4148a75686dc88059cf3 at {username='slave'} at 127.22.252.134:53955: session id = 5f8a0cfdfdca4148a75686dc88059cf3-205a0fda164d46a29a031070f1d0c369
I20240501 14:09:34.833915 30428 tablet_copy_source_session.cc:205] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:34.834874 1316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205a0fda164d46a29a031070f1d0c369. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.836968 32534 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:34.837014 1316 tablet_copy_client.cc:783] T 205a0fda164d46a29a031070f1d0c369 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:34.837162 1316 tablet_copy_client.cc:647] T 205a0fda164d46a29a031070f1d0c369 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:34.837913 30255 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:34.841025 329 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:34.849442 30386 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:34.850253 30125 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:34.850833 29993 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:34.862035 29872 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:34.863930 32400 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:34.870249 947 tablet.cc:1722] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:34.870574 1291 tablet.cc:1722] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:34.874812 1285 ts_tablet_manager.cc:902] T 53c51f3845c94f47b043621d98f70229 P a91029b90bf9415a89a2f70736759ad3: Initiating tablet copy from peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:34.875458 1297 tablet.cc:1722] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:34.876128 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=49c7b42b5f88482581798812bf953db3] alter complete (version 3)
I20240501 14:09:34.878144 1285 tablet_copy_client.cc:308] T 53c51f3845c94f47b043621d98f70229 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.132:34357
I20240501 14:09:34.884666 30428 tablet_copy_service.cc:140] P d1c217469ae4478ebd49e4ed9f7352d7: Received BeginTabletCopySession request for tablet 53c51f3845c94f47b043621d98f70229 from peer a91029b90bf9415a89a2f70736759ad3 ({username='slave'} at 127.22.252.133:43857)
I20240501 14:09:34.884775 30428 tablet_copy_service.cc:161] P d1c217469ae4478ebd49e4ed9f7352d7: Beginning new tablet copy session on tablet 53c51f3845c94f47b043621d98f70229 from peer a91029b90bf9415a89a2f70736759ad3 at {username='slave'} at 127.22.252.133:43857: session id = a91029b90bf9415a89a2f70736759ad3-53c51f3845c94f47b043621d98f70229
I20240501 14:09:34.885684 30428 tablet_copy_source_session.cc:205] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:34.886591 1285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53c51f3845c94f47b043621d98f70229. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.888582 1285 tablet_copy_client.cc:783] T 53c51f3845c94f47b043621d98f70229 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:34.888712 1285 tablet_copy_client.cc:647] T 53c51f3845c94f47b043621d98f70229 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:34.889796 1316 tablet_copy_client.cc:523] T 205a0fda164d46a29a031070f1d0c369 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:34.891112 1316 tablet_bootstrap.cc:492] T 205a0fda164d46a29a031070f1d0c369 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:34.895248 1285 tablet_copy_client.cc:523] T 53c51f3845c94f47b043621d98f70229 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:34.896422 1285 tablet_bootstrap.cc:492] T 53c51f3845c94f47b043621d98f70229 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:34.897359 32665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:34.915176 1316 tablet_bootstrap.cc:492] T 205a0fda164d46a29a031070f1d0c369 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:34.915423 1316 tablet_bootstrap.cc:492] T 205a0fda164d46a29a031070f1d0c369 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap complete.
I20240501 14:09:34.915577 1316 ts_tablet_manager.cc:1367] T 205a0fda164d46a29a031070f1d0c369 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.025s user 0.017s sys 0.004s
I20240501 14:09:34.915926 1316 raft_consensus.cc:348] T 205a0fda164d46a29a031070f1d0c369 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:34.916040 1316 raft_consensus.cc:729] T 205a0fda164d46a29a031070f1d0c369 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.916164 1316 consensus_queue.cc:260] T 205a0fda164d46a29a031070f1d0c369 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:34.916483 1316 ts_tablet_manager.cc:1398] T 205a0fda164d46a29a031070f1d0c369 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 14:09:34.920770 30428 tablet_copy_service.cc:342] P d1c217469ae4478ebd49e4ed9f7352d7: Request end of tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-205a0fda164d46a29a031070f1d0c369 received from {username='slave'} at 127.22.252.134:53955
I20240501 14:09:34.920850 30428 tablet_copy_service.cc:433] P d1c217469ae4478ebd49e4ed9f7352d7: ending tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-205a0fda164d46a29a031070f1d0c369 on tablet 205a0fda164d46a29a031070f1d0c369 with peer 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:09:34.922690 1285 tablet_bootstrap.cc:492] T 53c51f3845c94f47b043621d98f70229 P a91029b90bf9415a89a2f70736759ad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:34.922972 1285 tablet_bootstrap.cc:492] T 53c51f3845c94f47b043621d98f70229 P a91029b90bf9415a89a2f70736759ad3: Bootstrap complete.
I20240501 14:09:34.923138 1285 ts_tablet_manager.cc:1367] T 53c51f3845c94f47b043621d98f70229 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.027s user 0.019s sys 0.004s
I20240501 14:09:34.923540 1285 raft_consensus.cc:348] T 53c51f3845c94f47b043621d98f70229 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.923637 1285 raft_consensus.cc:729] T 53c51f3845c94f47b043621d98f70229 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.923751 1285 consensus_queue.cc:260] T 53c51f3845c94f47b043621d98f70229 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:34.923997 1285 ts_tablet_manager.cc:1398] T 53c51f3845c94f47b043621d98f70229 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:34.924417 30428 tablet_copy_service.cc:342] P d1c217469ae4478ebd49e4ed9f7352d7: Request end of tablet copy session a91029b90bf9415a89a2f70736759ad3-53c51f3845c94f47b043621d98f70229 received from {username='slave'} at 127.22.252.133:43857
I20240501 14:09:34.924476 30428 tablet_copy_service.cc:433] P d1c217469ae4478ebd49e4ed9f7352d7: ending tablet copy session a91029b90bf9415a89a2f70736759ad3-53c51f3845c94f47b043621d98f70229 on tablet 53c51f3845c94f47b043621d98f70229 with peer a91029b90bf9415a89a2f70736759ad3
I20240501 14:09:34.931435 32683 raft_consensus.cc:1203] T 354658866eda4a2da6a2871fe0a2769d P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.980000 29867 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:34.980125 29867 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:34.982841 29867 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 4b391050e0f4480faf30d2749f618cae
I20240501 14:09:34.983526 329 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b391050e0f4480faf30d2749f618cae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:34.983593 32665 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b391050e0f4480faf30d2749f618cae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:34.984081 32400 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b391050e0f4480faf30d2749f618cae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:34.984392 1310 tablet_replica.cc:330] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:34.984496 1310 raft_consensus.cc:2229] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.984594 1310 raft_consensus.cc:2258] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.985365 1310 ts_tablet_manager.cc:1875] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.987615 1310 ts_tablet_manager.cc:1888] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.987681 1310 log.cc:1196] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/4b391050e0f4480faf30d2749f618cae
I20240501 14:09:34.988960 1445 tablet_replica.cc:330] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:34.989096 1445 raft_consensus.cc:2229] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.989255 1445 raft_consensus.cc:2258] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.990041 1310 ts_tablet_manager.cc:1909] T 4b391050e0f4480faf30d2749f618cae P db935501e9104de5b15c8780681d6c38: Deleting consensus metadata
I20240501 14:09:34.991180 1445 ts_tablet_manager.cc:1875] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.992554 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet 4b391050e0f4480faf30d2749f618cae (table rebalancer_dynamic_table_new_name [id=49c7b42b5f88482581798812bf953db3]) successfully deleted
I20240501 14:09:34.996582 1445 ts_tablet_manager.cc:1888] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.996733 1445 log.cc:1196] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/4b391050e0f4480faf30d2749f618cae
I20240501 14:09:34.998536 1445 ts_tablet_manager.cc:1909] T 4b391050e0f4480faf30d2749f618cae P a91029b90bf9415a89a2f70736759ad3: Deleting consensus metadata
I20240501 14:09:34.999883 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 4b391050e0f4480faf30d2749f618cae (table rebalancer_dynamic_table_new_name [id=49c7b42b5f88482581798812bf953db3]) successfully deleted
I20240501 14:09:35.000666 1444 tablet_replica.cc:330] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:35.000895 1444 raft_consensus.cc:2229] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:35.001149 1444 raft_consensus.cc:2258] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.002348 1444 ts_tablet_manager.cc:1875] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.011782 1444 ts_tablet_manager.cc:1888] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:35.011914 1444 log.cc:1196] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/4b391050e0f4480faf30d2749f618cae
I20240501 14:09:35.014415 1444 ts_tablet_manager.cc:1909] T 4b391050e0f4480faf30d2749f618cae P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:09:35.015142 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 4b391050e0f4480faf30d2749f618cae (table rebalancer_dynamic_table_new_name [id=49c7b42b5f88482581798812bf953db3]) successfully deleted
I20240501 14:09:35.018697 30255 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:35.019419 1076 consensus_queue.cc:1035] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.021103 329 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:35.021503 29993 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:35.025164 32415 raft_consensus.cc:1203] T 1698ca3ec26741bb9b3131fe67c9682b P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:35.028702 32400 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:35.029460 32665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:35.037880 32534 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:35.038129 30386 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:35.047469 30125 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:35.048023 1076 consensus_queue.cc:1035] T 90a84c54afc241709d87597733f79df7 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.049264 1275 consensus_queue.cc:1035] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.074569 969 consensus_queue.cc:1035] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.084686 29869 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:35.090214 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 810e1e79d93a4cd9b4f4deef398ee653 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dfd3f624f08b4400bb7905c63a103997]), partition=
I20240501 14:09:35.090337 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 810e1e79d93a4cd9b4f4deef398ee653 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dfd3f624f08b4400bb7905c63a103997]), partition=
I20240501 14:09:35.090451 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810e1e79d93a4cd9b4f4deef398ee653. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.090579 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810e1e79d93a4cd9b4f4deef398ee653. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.090612 329 tablet_service.cc:1467] Processing CreateTablet for tablet 810e1e79d93a4cd9b4f4deef398ee653 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dfd3f624f08b4400bb7905c63a103997]), partition=
I20240501 14:09:35.090801 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810e1e79d93a4cd9b4f4deef398ee653. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.092135 1464 tablet_bootstrap.cc:492] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:35.092808 1465 tablet_bootstrap.cc:492] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:35.093216 1464 tablet_bootstrap.cc:654] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.093819 1465 tablet_bootstrap.cc:654] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.094177 1466 tablet_bootstrap.cc:492] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:35.095044 1464 tablet_bootstrap.cc:492] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:35.095165 1464 ts_tablet_manager.cc:1367] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:35.095321 1466 tablet_bootstrap.cc:654] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.095595 1464 raft_consensus.cc:348] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:35.095721 1464 raft_consensus.cc:374] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.095765 1464 raft_consensus.cc:729] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.095935 1464 consensus_queue.cc:260] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:35.096256 1464 ts_tablet_manager.cc:1398] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.097018 1465 tablet_bootstrap.cc:492] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:35.097103 1465 ts_tablet_manager.cc:1367] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:35.097477 1465 raft_consensus.cc:348] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:35.097560 1465 raft_consensus.cc:374] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.097591 1465 raft_consensus.cc:729] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.097712 1465 consensus_queue.cc:260] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:35.097975 1465 ts_tablet_manager.cc:1398] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.097915 1466 tablet_bootstrap.cc:492] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:35.098059 1466 ts_tablet_manager.cc:1367] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:35.098508 1466 raft_consensus.cc:348] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:35.099994 1466 raft_consensus.cc:374] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.100080 1466 raft_consensus.cc:729] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.100299 1466 consensus_queue.cc:260] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:35.100600 1466 ts_tablet_manager.cc:1398] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:35.136961 348 raft_consensus.cc:1203] T 3b6a526c2ed84e1780ec392ef39d09b0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:35.171521 32683 raft_consensus.cc:1203] T fc28b2028a9246b99e8887f03180d0c3 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:35.185999 29867 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 14:09:35.186123 29867 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 14:09:35.188316 29867 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 62b3902e073b42b18c3ff62bcdccb2cb
I20240501 14:09:35.188593 29867 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 9f15d8d22c324ff3a3315ecb2345553d
I20240501 14:09:35.188832 29867 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 4 replicas of tablet 8d5e1afe21654080b5aa3eb877e8f1d9
I20240501 14:09:35.188858 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 62b3902e073b42b18c3ff62bcdccb2cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:35.189049 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet 62b3902e073b42b18c3ff62bcdccb2cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:35.189092 30255 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f15d8d22c324ff3a3315ecb2345553d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:35.189241 1311 tablet_replica.cc:330] T 9f15d8d22c324ff3a3315ecb2345553d P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:35.189242 543 tablet_replica.cc:330] T 62b3902e073b42b18c3ff62bcdccb2cb P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:35.189359 543 raft_consensus.cc:2229] T 62b3902e073b42b18c3ff62bcdccb2cb P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.189359 1311 raft_consensus.cc:2229] T 9f15d8d22c324ff3a3315ecb2345553d P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.189399 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f15d8d22c324ff3a3315ecb2345553d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:35.189473 543 raft_consensus.cc:2258] T 62b3902e073b42b18c3ff62bcdccb2cb P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.189476 1311 raft_consensus.cc:2258] T 9f15d8d22c324ff3a3315ecb2345553d P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.189987 543 ts_tablet_manager.cc:1875] T 62b3902e073b42b18c3ff62bcdccb2cb P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.189996 1311 ts_tablet_manager.cc:1875] T 9f15d8d22c324ff3a3315ecb2345553d P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.192118 1311 ts_tablet_manager.cc:1888] T 9f15d8d22c324ff3a3315ecb2345553d P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:35.192118 543 ts_tablet_manager.cc:1888] T 62b3902e073b42b18c3ff62bcdccb2cb P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:35.192180 1311 log.cc:1196] T 9f15d8d22c324ff3a3315ecb2345553d P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/9f15d8d22c324ff3a3315ecb2345553d
I20240501 14:09:35.192180 543 log.cc:1196] T 62b3902e073b42b18c3ff62bcdccb2cb P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/62b3902e073b42b18c3ff62bcdccb2cb
I20240501 14:09:35.192361 545 tablet_replica.cc:330] T 62b3902e073b42b18c3ff62bcdccb2cb P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:35.192454 1311 ts_tablet_manager.cc:1909] T 9f15d8d22c324ff3a3315ecb2345553d P 1d0bb69611af4876a2b3b33c4766e13a: Deleting consensus metadata
I20240501 14:09:35.192502 545 raft_consensus.cc:2229] T 62b3902e073b42b18c3ff62bcdccb2cb P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.192587 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d5e1afe21654080b5aa3eb877e8f1d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:35.192633 545 raft_consensus.cc:2258] T 62b3902e073b42b18c3ff62bcdccb2cb P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.193044 32534 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d5e1afe21654080b5aa3eb877e8f1d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:35.193148 545 ts_tablet_manager.cc:1875] T 62b3902e073b42b18c3ff62bcdccb2cb P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.193351 1510 tablet_replica.cc:330] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:35.193384 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet 9f15d8d22c324ff3a3315ecb2345553d (table rebalance_test_table_10 [id=4fa8cc9c35c649efb9c70f46be41dd50]) successfully deleted
I20240501 14:09:35.193470 1510 raft_consensus.cc:2229] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.193562 1510 raft_consensus.cc:2258] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.193806 30386 tablet_service.cc:1514] Processing DeleteTablet for tablet 62b3902e073b42b18c3ff62bcdccb2cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:35.194003 30388 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f15d8d22c324ff3a3315ecb2345553d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:35.194149 546 tablet_replica.cc:330] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:35.194303 546 raft_consensus.cc:2229] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:35.194489 546 raft_consensus.cc:2258] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.195003 546 ts_tablet_manager.cc:1875] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.195308 545 ts_tablet_manager.cc:1888] T 62b3902e073b42b18c3ff62bcdccb2cb P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:35.195364 545 log.cc:1196] T 62b3902e073b42b18c3ff62bcdccb2cb P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/62b3902e073b42b18c3ff62bcdccb2cb
I20240501 14:09:35.195591 545 ts_tablet_manager.cc:1909] T 62b3902e073b42b18c3ff62bcdccb2cb P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:35.196126 545 tablet_replica.cc:330] T 9f15d8d22c324ff3a3315ecb2345553d P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:35.196198 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 62b3902e073b42b18c3ff62bcdccb2cb (table rebalance_test_table_10 [id=4fa8cc9c35c649efb9c70f46be41dd50]) successfully deleted
I20240501 14:09:35.197115 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d5e1afe21654080b5aa3eb877e8f1d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:35.197256 546 ts_tablet_manager.cc:1888] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:35.197329 546 log.cc:1196] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/62b3902e073b42b18c3ff62bcdccb2cb
I20240501 14:09:35.197630 546 ts_tablet_manager.cc:1909] T 62b3902e073b42b18c3ff62bcdccb2cb P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:35.197932 30386 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d5e1afe21654080b5aa3eb877e8f1d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:35.197995 543 ts_tablet_manager.cc:1909] T 62b3902e073b42b18c3ff62bcdccb2cb P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:35.198580 543 tablet_replica.cc:330] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:35.198658 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 62b3902e073b42b18c3ff62bcdccb2cb (table rebalance_test_table_10 [id=4fa8cc9c35c649efb9c70f46be41dd50]) successfully deleted
I20240501 14:09:35.198737 546 tablet_replica.cc:330] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:35.198849 546 raft_consensus.cc:2229] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:35.198928 545 raft_consensus.cc:2229] T 9f15d8d22c324ff3a3315ecb2345553d P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.199018 543 raft_consensus.cc:2229] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:35.199219 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 62b3902e073b42b18c3ff62bcdccb2cb (table rebalance_test_table_10 [id=4fa8cc9c35c649efb9c70f46be41dd50]) successfully deleted
I20240501 14:09:35.199209 546 raft_consensus.cc:2258] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.199395 545 raft_consensus.cc:2258] T 9f15d8d22c324ff3a3315ecb2345553d P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.199736 546 ts_tablet_manager.cc:1875] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.199921 543 raft_consensus.cc:2258] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.200379 545 ts_tablet_manager.cc:1875] T 9f15d8d22c324ff3a3315ecb2345553d P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.201064 543 ts_tablet_manager.cc:1875] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.201586 546 ts_tablet_manager.cc:1888] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:35.201658 546 log.cc:1196] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/9f15d8d22c324ff3a3315ecb2345553d
I20240501 14:09:35.201911 546 ts_tablet_manager.cc:1909] T 9f15d8d22c324ff3a3315ecb2345553d P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:35.202428 546 tablet_replica.cc:330] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:35.202530 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 9f15d8d22c324ff3a3315ecb2345553d (table rebalance_test_table_10 [id=4fa8cc9c35c649efb9c70f46be41dd50]) successfully deleted
I20240501 14:09:35.202567 546 raft_consensus.cc:2229] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.202741 546 raft_consensus.cc:2258] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.203217 546 ts_tablet_manager.cc:1875] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.203666 545 ts_tablet_manager.cc:1888] T 9f15d8d22c324ff3a3315ecb2345553d P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:35.203742 545 log.cc:1196] T 9f15d8d22c324ff3a3315ecb2345553d P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/9f15d8d22c324ff3a3315ecb2345553d
I20240501 14:09:35.203998 545 ts_tablet_manager.cc:1909] T 9f15d8d22c324ff3a3315ecb2345553d P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:35.204576 545 tablet_replica.cc:330] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:35.204666 1510 ts_tablet_manager.cc:1875] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.204684 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 9f15d8d22c324ff3a3315ecb2345553d (table rebalance_test_table_10 [id=4fa8cc9c35c649efb9c70f46be41dd50]) successfully deleted
I20240501 14:09:35.205078 546 ts_tablet_manager.cc:1888] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:35.205152 546 log.cc:1196] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/8d5e1afe21654080b5aa3eb877e8f1d9
I20240501 14:09:35.205444 546 ts_tablet_manager.cc:1909] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:35.205996 543 ts_tablet_manager.cc:1888] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:35.206073 543 log.cc:1196] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/8d5e1afe21654080b5aa3eb877e8f1d9
I20240501 14:09:35.206180 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 8d5e1afe21654080b5aa3eb877e8f1d9 (table rebalance_test_table_10 [id=4fa8cc9c35c649efb9c70f46be41dd50]) successfully deleted
I20240501 14:09:35.206338 543 ts_tablet_manager.cc:1909] T 8d5e1afe21654080b5aa3eb877e8f1d9 P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:35.206624 1510 ts_tablet_manager.cc:1888] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:35.206699 1510 log.cc:1196] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/8d5e1afe21654080b5aa3eb877e8f1d9
I20240501 14:09:35.206929 545 raft_consensus.cc:2229] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.206957 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 8d5e1afe21654080b5aa3eb877e8f1d9 (table rebalance_test_table_10 [id=4fa8cc9c35c649efb9c70f46be41dd50]) successfully deleted
I20240501 14:09:35.207010 1510 ts_tablet_manager.cc:1909] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:35.207136 545 raft_consensus.cc:2258] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.207516 545 ts_tablet_manager.cc:1875] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.207619 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet 8d5e1afe21654080b5aa3eb877e8f1d9 (table rebalance_test_table_10 [id=4fa8cc9c35c649efb9c70f46be41dd50]) successfully deleted
I20240501 14:09:35.208900 545 ts_tablet_manager.cc:1888] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:35.208958 545 log.cc:1196] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/8d5e1afe21654080b5aa3eb877e8f1d9
I20240501 14:09:35.209141 545 ts_tablet_manager.cc:1909] T 8d5e1afe21654080b5aa3eb877e8f1d9 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:35.209614 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 8d5e1afe21654080b5aa3eb877e8f1d9 (table rebalance_test_table_10 [id=4fa8cc9c35c649efb9c70f46be41dd50]) successfully deleted
I20240501 14:09:35.227838 32553 raft_consensus.cc:1203] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:35.231652 32415 raft_consensus.cc:1203] T fb36e198bcf34ef68a9595d7cbe93e80 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:35.234623 1138 raft_consensus.cc:482] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:35.234730 1138 raft_consensus.cc:504] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:35.235172 1138 leader_election.cc:288] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:35.235464 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810e1e79d93a4cd9b4f4deef398ee653" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:35.235601 32415 raft_consensus.cc:2454] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:35.235878 32752 leader_election.cc:302] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:35.236043 1138 raft_consensus.cc:2781] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:35.236089 1138 raft_consensus.cc:482] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:35.236124 1138 raft_consensus.cc:3037] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.237179 1138 raft_consensus.cc:504] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:35.237540 1138 leader_election.cc:288] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:35.237921 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810e1e79d93a4cd9b4f4deef398ee653" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:35.238039 32415 raft_consensus.cc:3037] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.239003 32415 raft_consensus.cc:2454] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:35.239305 32752 leader_election.cc:302] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:35.239332 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810e1e79d93a4cd9b4f4deef398ee653" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:35.239372 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810e1e79d93a4cd9b4f4deef398ee653" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:35.239444 32553 raft_consensus.cc:2454] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:35.239501 1138 raft_consensus.cc:2781] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:35.239583 1138 raft_consensus.cc:686] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:35.239696 1138 consensus_queue.cc:237] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:35.241218 29867 catalog_manager.cc:5484] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:35.262357 32554 raft_consensus.cc:1203] T 205a0fda164d46a29a031070f1d0c369 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:35.268764 32534 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:35.271165 32400 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:35.271250 329 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:35.271497 348 raft_consensus.cc:1203] T 18918995fb03421b9dfb4d05369c1ce8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:35.276501 29993 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:35.276721 30386 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:35.282847 32665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:35.286149 1253 consensus_queue.cc:1035] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.289408 30255 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:35.291023 30125 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:35.292013 562 consensus_queue.cc:1035] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.351867 29867 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:35.354763 30406 consensus_queue.cc:237] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:35.356407 32683 raft_consensus.cc:1261] T dd2fd47ec6684cb28d2f4678d2e90e75 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.356752 30277 raft_consensus.cc:1261] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.357033 32554 raft_consensus.cc:3037] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.357192 731 consensus_queue.cc:1035] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.357511 731 consensus_queue.cc:1035] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.357550 32415 raft_consensus.cc:1261] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:35.358036 1138 consensus_queue.cc:1035] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.358201 32554 raft_consensus.cc:1261] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:35.358742 29867 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:35.358974 1138 consensus_queue.cc:1035] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.361560 995 raft_consensus.cc:2932] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:35.362222 30264 raft_consensus.cc:2932] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:35.362982 947 tablet.cc:1722] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:35.363301 947 tablet.cc:1620] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:35.363467 29872 catalog_manager.cc:5576] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497) does not have the latest schema for tablet 810e1e79d93a4cd9b4f4deef398ee653 (table rebalancer_dynamic_table [id=dfd3f624f08b4400bb7905c63a103997]). Expected version 1 got 0
I20240501 14:09:35.365661 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 776d6722a14345f490f69f90d80d6a6a (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=d23b9f10156c4074a707699cf10e3b55]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.365924 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776d6722a14345f490f69f90d80d6a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.366438 329 tablet_service.cc:1467] Processing CreateTablet for tablet 776d6722a14345f490f69f90d80d6a6a (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=d23b9f10156c4074a707699cf10e3b55]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.366701 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776d6722a14345f490f69f90d80d6a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.366736 328 tablet_service.cc:1467] Processing CreateTablet for tablet f2541122e05f45c2843d06c5342c8d01 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=d23b9f10156c4074a707699cf10e3b55]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.366925 328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2541122e05f45c2843d06c5342c8d01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.368510 30386 tablet_service.cc:1467] Processing CreateTablet for tablet bf97db677b9d426486f0fc6d751d667c (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=d23b9f10156c4074a707699cf10e3b55]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.368757 30386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf97db677b9d426486f0fc6d751d667c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.368561 1549 tablet.cc:1722] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:35.368991 1549 tablet.cc:1620] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:35.368963 29993 tablet_service.cc:1467] Processing CreateTablet for tablet f2541122e05f45c2843d06c5342c8d01 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=d23b9f10156c4074a707699cf10e3b55]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.369235 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2541122e05f45c2843d06c5342c8d01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.369555 1291 tablet.cc:1722] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:35.369793 1291 tablet.cc:1620] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:35.370878 1550 tablet_bootstrap.cc:492] T bf97db677b9d426486f0fc6d751d667c P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:35.371058 1465 tablet_bootstrap.cc:492] T 776d6722a14345f490f69f90d80d6a6a P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:35.371704 1550 tablet_bootstrap.cc:654] T bf97db677b9d426486f0fc6d751d667c P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.371675 29995 tablet_service.cc:1467] Processing CreateTablet for tablet bf97db677b9d426486f0fc6d751d667c (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=d23b9f10156c4074a707699cf10e3b55]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.371874 1465 tablet_bootstrap.cc:654] T 776d6722a14345f490f69f90d80d6a6a P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.371878 29995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf97db677b9d426486f0fc6d751d667c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.372510 32402 tablet_service.cc:1467] Processing CreateTablet for tablet f2541122e05f45c2843d06c5342c8d01 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=d23b9f10156c4074a707699cf10e3b55]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.372699 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 776d6722a14345f490f69f90d80d6a6a (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=d23b9f10156c4074a707699cf10e3b55]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.372774 32402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2541122e05f45c2843d06c5342c8d01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.372956 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776d6722a14345f490f69f90d80d6a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.373091 1466 tablet_bootstrap.cc:492] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:35.373373 1465 tablet_bootstrap.cc:492] T 776d6722a14345f490f69f90d80d6a6a P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:35.373440 1465 ts_tablet_manager.cc:1367] T 776d6722a14345f490f69f90d80d6a6a P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:35.373731 1466 tablet_bootstrap.cc:654] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.373854 1465 raft_consensus.cc:348] T 776d6722a14345f490f69f90d80d6a6a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:35.374006 1465 raft_consensus.cc:374] T 776d6722a14345f490f69f90d80d6a6a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.374037 1465 raft_consensus.cc:729] T 776d6722a14345f490f69f90d80d6a6a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.374137 1465 consensus_queue.cc:260] T 776d6722a14345f490f69f90d80d6a6a P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:35.374472 1465 ts_tablet_manager.cc:1398] T 776d6722a14345f490f69f90d80d6a6a P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:35.374625 1466 tablet_bootstrap.cc:492] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:35.374693 1466 ts_tablet_manager.cc:1367] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:35.374902 1464 tablet_bootstrap.cc:492] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:35.375089 1466 raft_consensus.cc:348] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:35.375172 1466 raft_consensus.cc:374] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.375198 1466 raft_consensus.cc:729] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.375321 1466 consensus_queue.cc:260] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:35.375566 1466 ts_tablet_manager.cc:1398] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.375794 1464 tablet_bootstrap.cc:654] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.378922 1466 tablet_bootstrap.cc:492] T 776d6722a14345f490f69f90d80d6a6a P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:35.378875 29873 catalog_manager.cc:5484] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:35.379359 29871 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=dfd3f624f08b4400bb7905c63a103997] alter complete (version 1)
I20240501 14:09:35.379773 1466 tablet_bootstrap.cc:654] T 776d6722a14345f490f69f90d80d6a6a P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.380715 1551 tablet_bootstrap.cc:492] T f2541122e05f45c2843d06c5342c8d01 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:35.380535 32683 raft_consensus.cc:2932] T dd2fd47ec6684cb28d2f4678d2e90e75 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:35.381886 1551 tablet_bootstrap.cc:654] T f2541122e05f45c2843d06c5342c8d01 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.383301 1551 tablet_bootstrap.cc:492] T f2541122e05f45c2843d06c5342c8d01 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:35.383383 1551 ts_tablet_manager.cc:1367] T f2541122e05f45c2843d06c5342c8d01 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:35.383446 1466 tablet_bootstrap.cc:492] T 776d6722a14345f490f69f90d80d6a6a P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:35.383509 1466 ts_tablet_manager.cc:1367] T 776d6722a14345f490f69f90d80d6a6a P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240501 14:09:35.383795 1551 raft_consensus.cc:348] T f2541122e05f45c2843d06c5342c8d01 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:35.383886 1551 raft_consensus.cc:374] T f2541122e05f45c2843d06c5342c8d01 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.383924 1551 raft_consensus.cc:729] T f2541122e05f45c2843d06c5342c8d01 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.383894 1466 raft_consensus.cc:348] T 776d6722a14345f490f69f90d80d6a6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:35.383989 1466 raft_consensus.cc:374] T 776d6722a14345f490f69f90d80d6a6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.384019 1466 raft_consensus.cc:729] T 776d6722a14345f490f69f90d80d6a6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.384042 1551 consensus_queue.cc:260] T f2541122e05f45c2843d06c5342c8d01 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:35.384117 1466 consensus_queue.cc:260] T 776d6722a14345f490f69f90d80d6a6a P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:35.384342 1551 ts_tablet_manager.cc:1398] T f2541122e05f45c2843d06c5342c8d01 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.384428 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet dd2fd47ec6684cb28d2f4678d2e90e75 with delete_type TABLET_DATA_TOMBSTONED (TS d35c16dcb506426993811bb2519d7db3 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:35.384479 1551 tablet_bootstrap.cc:492] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:35.384641 543 tablet_replica.cc:330] T dd2fd47ec6684cb28d2f4678d2e90e75 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:35.384770 543 raft_consensus.cc:2229] T dd2fd47ec6684cb28d2f4678d2e90e75 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.384912 543 raft_consensus.cc:2258] T dd2fd47ec6684cb28d2f4678d2e90e75 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.385154 1551 tablet_bootstrap.cc:654] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.385197 1466 ts_tablet_manager.cc:1398] T 776d6722a14345f490f69f90d80d6a6a P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:35.385409 543 ts_tablet_manager.cc:1875] T dd2fd47ec6684cb28d2f4678d2e90e75 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:35.386687 1551 tablet_bootstrap.cc:492] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:35.386759 1551 ts_tablet_manager.cc:1367] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:35.387070 29872 catalog_manager.cc:5576] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265) does not have the latest schema for tablet 810e1e79d93a4cd9b4f4deef398ee653 (table rebalancer_dynamic_table [id=dfd3f624f08b4400bb7905c63a103997]). Expected version 1 got 0
I20240501 14:09:35.387092 1551 raft_consensus.cc:348] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:35.387217 1551 raft_consensus.cc:374] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.387254 1551 raft_consensus.cc:729] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.387388 1551 consensus_queue.cc:260] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:35.387522 543 ts_tablet_manager.cc:1888] T dd2fd47ec6684cb28d2f4678d2e90e75 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:35.387589 543 log.cc:1196] T dd2fd47ec6684cb28d2f4678d2e90e75 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/dd2fd47ec6684cb28d2f4678d2e90e75
I20240501 14:09:35.387636 1551 ts_tablet_manager.cc:1398] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.388120 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet dd2fd47ec6684cb28d2f4678d2e90e75 (table rebalance_test_table_17 [id=9a4b9a0a2fe64ab2a64d6679185c2a33]) successfully deleted
I20240501 14:09:35.389092 32534 tablet_service.cc:1467] Processing CreateTablet for tablet bf97db677b9d426486f0fc6d751d667c (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=d23b9f10156c4074a707699cf10e3b55]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.389334 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf97db677b9d426486f0fc6d751d667c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.390101 1550 tablet_bootstrap.cc:492] T bf97db677b9d426486f0fc6d751d667c P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:35.390208 1550 ts_tablet_manager.cc:1367] T bf97db677b9d426486f0fc6d751d667c P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.000s
I20240501 14:09:35.390678 1550 raft_consensus.cc:348] T bf97db677b9d426486f0fc6d751d667c P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:35.390785 1550 raft_consensus.cc:374] T bf97db677b9d426486f0fc6d751d667c P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.390821 1550 raft_consensus.cc:729] T bf97db677b9d426486f0fc6d751d667c P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.390973 1550 consensus_queue.cc:260] T bf97db677b9d426486f0fc6d751d667c P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:35.391271 1550 ts_tablet_manager.cc:1398] T bf97db677b9d426486f0fc6d751d667c P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:35.393365 1465 tablet_bootstrap.cc:492] T bf97db677b9d426486f0fc6d751d667c P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:35.395123 1465 tablet_bootstrap.cc:654] T bf97db677b9d426486f0fc6d751d667c P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.397030 1513 raft_consensus.cc:482] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:35.397116 1513 raft_consensus.cc:504] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:35.397339 1464 tablet_bootstrap.cc:492] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:35.397423 1464 ts_tablet_manager.cc:1367] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.000s
I20240501 14:09:35.397491 1513 leader_election.cc:288] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:35.397733 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf97db677b9d426486f0fc6d751d667c" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:35.397836 30406 raft_consensus.cc:2454] T bf97db677b9d426486f0fc6d751d667c P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:35.397895 1464 raft_consensus.cc:348] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:35.398006 1464 raft_consensus.cc:374] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.398044 1464 raft_consensus.cc:729] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.398165 1464 consensus_queue.cc:260] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:35.398491 1464 ts_tablet_manager.cc:1398] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:35.398837 1464 tablet_bootstrap.cc:492] T f2541122e05f45c2843d06c5342c8d01 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:35.398891 29948 leader_election.cc:302] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:35.399057 1513 raft_consensus.cc:2781] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:35.399106 1513 raft_consensus.cc:482] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:35.399150 1513 raft_consensus.cc:3037] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.399914 1464 tablet_bootstrap.cc:654] T f2541122e05f45c2843d06c5342c8d01 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.400285 1513 raft_consensus.cc:504] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:35.400496 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf97db677b9d426486f0fc6d751d667c" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:35.400646 1513 leader_election.cc:288] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
W20240501 14:09:35.401203 29951 leader_election.cc:341] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:35.401680 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf97db677b9d426486f0fc6d751d667c" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
W20240501 14:09:35.402072 29951 leader_election.cc:341] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:35.402381 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf97db677b9d426486f0fc6d751d667c" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:35.402477 30406 raft_consensus.cc:3037] T bf97db677b9d426486f0fc6d751d667c P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.403426 30406 raft_consensus.cc:2454] T bf97db677b9d426486f0fc6d751d667c P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:35.403628 1464 tablet_bootstrap.cc:492] T f2541122e05f45c2843d06c5342c8d01 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:35.403708 1464 ts_tablet_manager.cc:1367] T f2541122e05f45c2843d06c5342c8d01 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:35.403843 29948 leader_election.cc:302] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters: 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:09:35.403982 1513 raft_consensus.cc:2781] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:35.404062 1513 raft_consensus.cc:686] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:35.404096 640 raft_consensus.cc:482] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:35.404076 1464 raft_consensus.cc:348] T f2541122e05f45c2843d06c5342c8d01 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:35.404160 1464 raft_consensus.cc:374] T f2541122e05f45c2843d06c5342c8d01 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.404158 640 raft_consensus.cc:504] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:35.404209 1464 raft_consensus.cc:729] T f2541122e05f45c2843d06c5342c8d01 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.404175 1513 consensus_queue.cc:237] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:35.404384 1464 consensus_queue.cc:260] T f2541122e05f45c2843d06c5342c8d01 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:35.404508 640 leader_election.cc:288] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:35.404976 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2541122e05f45c2843d06c5342c8d01" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:35.405090 32415 raft_consensus.cc:2454] T f2541122e05f45c2843d06c5342c8d01 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:35.405213 1464 ts_tablet_manager.cc:1398] T f2541122e05f45c2843d06c5342c8d01 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.405344 32752 leader_election.cc:302] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:35.405556 640 raft_consensus.cc:2781] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:35.405606 640 raft_consensus.cc:482] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:35.405652 640 raft_consensus.cc:3037] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.406327 29872 catalog_manager.cc:5484] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:35.406450 640 raft_consensus.cc:504] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:35.406772 640 leader_election.cc:288] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:35.407075 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2541122e05f45c2843d06c5342c8d01" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:35.407182 30014 raft_consensus.cc:2454] T f2541122e05f45c2843d06c5342c8d01 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:35.407717 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2541122e05f45c2843d06c5342c8d01" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:35.407814 32415 raft_consensus.cc:3037] T f2541122e05f45c2843d06c5342c8d01 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.408039 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2541122e05f45c2843d06c5342c8d01" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:35.408135 30014 raft_consensus.cc:3037] T f2541122e05f45c2843d06c5342c8d01 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.408756 32415 raft_consensus.cc:2454] T f2541122e05f45c2843d06c5342c8d01 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:35.408901 30014 raft_consensus.cc:2454] T f2541122e05f45c2843d06c5342c8d01 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:35.409080 32752 leader_election.cc:302] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:35.409404 640 raft_consensus.cc:2781] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:35.409488 640 raft_consensus.cc:686] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:35.409597 640 consensus_queue.cc:237] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:35.412879 29872 catalog_manager.cc:5484] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:35.428620 1465 tablet_bootstrap.cc:492] T bf97db677b9d426486f0fc6d751d667c P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:35.428727 1465 ts_tablet_manager.cc:1367] T bf97db677b9d426486f0fc6d751d667c P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.002s
I20240501 14:09:35.429127 1465 raft_consensus.cc:348] T bf97db677b9d426486f0fc6d751d667c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:35.429248 1465 raft_consensus.cc:374] T bf97db677b9d426486f0fc6d751d667c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.429277 1465 raft_consensus.cc:729] T bf97db677b9d426486f0fc6d751d667c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.429387 1465 consensus_queue.cc:260] T bf97db677b9d426486f0fc6d751d667c P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:35.429628 1465 ts_tablet_manager.cc:1398] T bf97db677b9d426486f0fc6d751d667c P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:35.457144 32415 raft_consensus.cc:1203] T 53c51f3845c94f47b043621d98f70229 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:35.489025 29867 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:35.493710 947 tablet.cc:1722] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:35.494040 947 tablet.cc:1620] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:35.494002 1549 tablet.cc:1722] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:35.494284 1549 tablet.cc:1620] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:35.494223 1291 tablet.cc:1722] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:35.494529 1291 tablet.cc:1620] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:35.498800 29870 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=dfd3f624f08b4400bb7905c63a103997] alter complete (version 2)
I20240501 14:09:35.562894 562 raft_consensus.cc:482] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:35.563004 562 raft_consensus.cc:504] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:35.563344 562 leader_election.cc:288] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:35.563634 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776d6722a14345f490f69f90d80d6a6a" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:35.563666 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776d6722a14345f490f69f90d80d6a6a" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:35.563764 348 raft_consensus.cc:2454] T 776d6722a14345f490f69f90d80d6a6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:35.563776 32554 raft_consensus.cc:2454] T 776d6722a14345f490f69f90d80d6a6a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:35.564019 32359 leader_election.cc:302] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:35.564172 562 raft_consensus.cc:2781] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:35.564252 562 raft_consensus.cc:482] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:35.564304 562 raft_consensus.cc:3037] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.565086 562 raft_consensus.cc:504] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:35.565392 562 leader_election.cc:288] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:35.565615 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776d6722a14345f490f69f90d80d6a6a" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:35.565716 348 raft_consensus.cc:3037] T 776d6722a14345f490f69f90d80d6a6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.565721 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776d6722a14345f490f69f90d80d6a6a" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:35.565827 32554 raft_consensus.cc:3037] T 776d6722a14345f490f69f90d80d6a6a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.566548 348 raft_consensus.cc:2454] T 776d6722a14345f490f69f90d80d6a6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:35.566540 32554 raft_consensus.cc:2454] T 776d6722a14345f490f69f90d80d6a6a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:35.566807 32359 leader_election.cc:302] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:35.566951 562 raft_consensus.cc:2781] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:35.567029 562 raft_consensus.cc:686] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:35.567155 562 consensus_queue.cc:237] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:35.568751 29869 catalog_manager.cc:5484] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:35.595504 32534 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:35.601547 29869 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:35.603924 329 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:35.616911 30255 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:35.616849 947 tablet.cc:1722] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:35.621599 30125 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:35.625523 1291 tablet.cc:1722] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:35.626134 1549 tablet.cc:1722] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:35.627739 29995 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:35.632651 29870 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=dfd3f624f08b4400bb7905c63a103997] alter complete (version 3)
I20240501 14:09:35.632934 30386 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:35.633777 32665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:35.638252 32402 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:35.694972 29869 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 14:09:35.695118 29869 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 14:09:35.696547 30406 consensus_queue.cc:237] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:35.697749 348 raft_consensus.cc:1261] T 940c8f54339f4a32be2942682a41c685 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.697770 29869 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet aa03df5f44864c9eb0a642f0481def39
I20240501 14:09:35.698076 29869 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 4 replicas of tablet 354658866eda4a2da6a2871fe0a2769d
I20240501 14:09:35.698100 30146 raft_consensus.cc:1261] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.698447 29869 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 92cac7a9b26444e3916d4d7e5dd0032b
I20240501 14:09:35.698501 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet aa03df5f44864c9eb0a642f0481def39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:35.698514 995 consensus_queue.cc:1035] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.698734 30386 tablet_service.cc:1514] Processing DeleteTablet for tablet 354658866eda4a2da6a2871fe0a2769d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:35.698836 543 tablet_replica.cc:330] T aa03df5f44864c9eb0a642f0481def39 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:35.698880 546 tablet_replica.cc:330] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:35.698989 543 raft_consensus.cc:2229] T aa03df5f44864c9eb0a642f0481def39 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.699018 546 raft_consensus.cc:2229] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:35.699113 543 raft_consensus.cc:2258] T aa03df5f44864c9eb0a642f0481def39 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.699200 546 raft_consensus.cc:2258] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.699656 546 ts_tablet_manager.cc:1875] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.699656 543 ts_tablet_manager.cc:1875] T aa03df5f44864c9eb0a642f0481def39 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.699641 29995 tablet_service.cc:1514] Processing DeleteTablet for tablet aa03df5f44864c9eb0a642f0481def39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:35.699878 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet 354658866eda4a2da6a2871fe0a2769d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:35.699929 30255 tablet_service.cc:1514] Processing DeleteTablet for tablet aa03df5f44864c9eb0a642f0481def39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:35.700018 545 tablet_replica.cc:330] T aa03df5f44864c9eb0a642f0481def39 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:35.700155 545 raft_consensus.cc:2229] T aa03df5f44864c9eb0a642f0481def39 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.700183 30256 tablet_service.cc:1514] Processing DeleteTablet for tablet 92cac7a9b26444e3916d4d7e5dd0032b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:35.700309 545 raft_consensus.cc:2258] T aa03df5f44864c9eb0a642f0481def39 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.700314 29994 tablet_service.cc:1514] Processing DeleteTablet for tablet 92cac7a9b26444e3916d4d7e5dd0032b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:35.700762 545 ts_tablet_manager.cc:1875] T aa03df5f44864c9eb0a642f0481def39 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.700937 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 354658866eda4a2da6a2871fe0a2769d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:35.701424 30386 tablet_service.cc:1514] Processing DeleteTablet for tablet 92cac7a9b26444e3916d4d7e5dd0032b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:35.702095 543 ts_tablet_manager.cc:1888] T aa03df5f44864c9eb0a642f0481def39 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:35.702165 543 log.cc:1196] T aa03df5f44864c9eb0a642f0481def39 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/aa03df5f44864c9eb0a642f0481def39
I20240501 14:09:35.702432 543 ts_tablet_manager.cc:1909] T aa03df5f44864c9eb0a642f0481def39 P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:35.702694 545 ts_tablet_manager.cc:1888] T aa03df5f44864c9eb0a642f0481def39 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:35.702759 545 log.cc:1196] T aa03df5f44864c9eb0a642f0481def39 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/aa03df5f44864c9eb0a642f0481def39
I20240501 14:09:35.703006 543 tablet_replica.cc:330] T 354658866eda4a2da6a2871fe0a2769d P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:35.703035 545 ts_tablet_manager.cc:1909] T aa03df5f44864c9eb0a642f0481def39 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:35.703123 543 raft_consensus.cc:2229] T 354658866eda4a2da6a2871fe0a2769d P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.703258 543 raft_consensus.cc:2258] T 354658866eda4a2da6a2871fe0a2769d P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.703529 545 tablet_replica.cc:330] T 92cac7a9b26444e3916d4d7e5dd0032b P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:35.703616 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet aa03df5f44864c9eb0a642f0481def39 (table rebalance_test_table_12 [id=b37dbfc47a574f218dd59bef604c1bc1]) successfully deleted
I20240501 14:09:35.703647 545 raft_consensus.cc:2229] T 92cac7a9b26444e3916d4d7e5dd0032b P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.703716 543 ts_tablet_manager.cc:1875] T 354658866eda4a2da6a2871fe0a2769d P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.703742 546 ts_tablet_manager.cc:1888] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:35.703763 545 raft_consensus.cc:2258] T 92cac7a9b26444e3916d4d7e5dd0032b P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.703799 546 log.cc:1196] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/354658866eda4a2da6a2871fe0a2769d
I20240501 14:09:35.704061 546 ts_tablet_manager.cc:1909] T 354658866eda4a2da6a2871fe0a2769d P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:35.704162 545 ts_tablet_manager.cc:1875] T 92cac7a9b26444e3916d4d7e5dd0032b P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.704357 32665 tablet_service.cc:1514] Processing DeleteTablet for tablet 354658866eda4a2da6a2871fe0a2769d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:35.704656 1635 tablet_replica.cc:330] T 354658866eda4a2da6a2871fe0a2769d P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:35.704689 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet aa03df5f44864c9eb0a642f0481def39 (table rebalance_test_table_12 [id=b37dbfc47a574f218dd59bef604c1bc1]) successfully deleted
I20240501 14:09:35.704843 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 354658866eda4a2da6a2871fe0a2769d (table rebalance_test_table_12 [id=b37dbfc47a574f218dd59bef604c1bc1]) successfully deleted
I20240501 14:09:35.704872 1635 raft_consensus.cc:2229] T 354658866eda4a2da6a2871fe0a2769d P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.705075 1635 raft_consensus.cc:2258] T 354658866eda4a2da6a2871fe0a2769d P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.705076 995 consensus_queue.cc:1035] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.705128 546 tablet_replica.cc:330] T 92cac7a9b26444e3916d4d7e5dd0032b P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:35.705314 546 raft_consensus.cc:2229] T 92cac7a9b26444e3916d4d7e5dd0032b P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.705654 1635 ts_tablet_manager.cc:1875] T 354658866eda4a2da6a2871fe0a2769d P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.705439 546 raft_consensus.cc:2258] T 92cac7a9b26444e3916d4d7e5dd0032b P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.706247 546 ts_tablet_manager.cc:1875] T 92cac7a9b26444e3916d4d7e5dd0032b P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.706812 545 ts_tablet_manager.cc:1888] T 92cac7a9b26444e3916d4d7e5dd0032b P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:35.706944 545 log.cc:1196] T 92cac7a9b26444e3916d4d7e5dd0032b P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/92cac7a9b26444e3916d4d7e5dd0032b
I20240501 14:09:35.707017 543 ts_tablet_manager.cc:1888] T 354658866eda4a2da6a2871fe0a2769d P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:35.707168 543 log.cc:1196] T 354658866eda4a2da6a2871fe0a2769d P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/354658866eda4a2da6a2871fe0a2769d
I20240501 14:09:35.707293 545 ts_tablet_manager.cc:1909] T 92cac7a9b26444e3916d4d7e5dd0032b P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:35.707527 543 ts_tablet_manager.cc:1909] T 354658866eda4a2da6a2871fe0a2769d P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:35.708144 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 354658866eda4a2da6a2871fe0a2769d (table rebalance_test_table_12 [id=b37dbfc47a574f218dd59bef604c1bc1]) successfully deleted
I20240501 14:09:35.708220 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 92cac7a9b26444e3916d4d7e5dd0032b (table rebalance_test_table_12 [id=b37dbfc47a574f218dd59bef604c1bc1]) successfully deleted
I20240501 14:09:35.708293 546 ts_tablet_manager.cc:1888] T 92cac7a9b26444e3916d4d7e5dd0032b P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:35.708357 546 log.cc:1196] T 92cac7a9b26444e3916d4d7e5dd0032b P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/92cac7a9b26444e3916d4d7e5dd0032b
I20240501 14:09:35.708456 545 tablet_replica.cc:330] T 354658866eda4a2da6a2871fe0a2769d P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:35.708498 1635 ts_tablet_manager.cc:1888] T 354658866eda4a2da6a2871fe0a2769d P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:35.708604 1635 log.cc:1196] T 354658866eda4a2da6a2871fe0a2769d P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/354658866eda4a2da6a2871fe0a2769d
I20240501 14:09:35.708724 546 ts_tablet_manager.cc:1909] T 92cac7a9b26444e3916d4d7e5dd0032b P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:35.708925 1635 ts_tablet_manager.cc:1909] T 354658866eda4a2da6a2871fe0a2769d P db935501e9104de5b15c8780681d6c38: Deleting consensus metadata
I20240501 14:09:35.708879 995 raft_consensus.cc:2932] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:35.709232 348 raft_consensus.cc:2932] T 940c8f54339f4a32be2942682a41c685 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:35.709645 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet 354658866eda4a2da6a2871fe0a2769d (table rebalance_test_table_12 [id=b37dbfc47a574f218dd59bef604c1bc1]) successfully deleted
I20240501 14:09:35.708590 545 raft_consensus.cc:2229] T 354658866eda4a2da6a2871fe0a2769d P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.710305 545 raft_consensus.cc:2258] T 354658866eda4a2da6a2871fe0a2769d P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.710883 545 ts_tablet_manager.cc:1875] T 354658866eda4a2da6a2871fe0a2769d P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.712677 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 92cac7a9b26444e3916d4d7e5dd0032b (table rebalance_test_table_12 [id=b37dbfc47a574f218dd59bef604c1bc1]) successfully deleted
I20240501 14:09:35.712777 29869 catalog_manager.cc:5484] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:35.714520 30146 raft_consensus.cc:2932] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:35.714890 545 ts_tablet_manager.cc:1888] T 354658866eda4a2da6a2871fe0a2769d P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:35.715003 545 log.cc:1196] T 354658866eda4a2da6a2871fe0a2769d P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/354658866eda4a2da6a2871fe0a2769d
I20240501 14:09:35.715236 30256 tablet_service.cc:1514] Processing DeleteTablet for tablet 940c8f54339f4a32be2942682a41c685 with delete_type TABLET_DATA_TOMBSTONED (TS 1d0bb69611af4876a2b3b33c4766e13a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:35.715339 545 ts_tablet_manager.cc:1909] T 354658866eda4a2da6a2871fe0a2769d P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:35.716173 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 354658866eda4a2da6a2871fe0a2769d (table rebalance_test_table_12 [id=b37dbfc47a574f218dd59bef604c1bc1]) successfully deleted
I20240501 14:09:35.716676 1634 tablet_replica.cc:330] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:35.716858 1634 raft_consensus.cc:2229] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:35.717145 1634 raft_consensus.cc:2258] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.717841 1634 ts_tablet_manager.cc:1875] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.719430 1634 ts_tablet_manager.cc:1888] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:35.719506 1634 log.cc:1196] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/aa03df5f44864c9eb0a642f0481def39
I20240501 14:09:35.719707 1634 ts_tablet_manager.cc:1909] T aa03df5f44864c9eb0a642f0481def39 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting consensus metadata
I20240501 14:09:35.720394 1634 tablet_replica.cc:330] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:35.720469 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet aa03df5f44864c9eb0a642f0481def39 (table rebalance_test_table_12 [id=b37dbfc47a574f218dd59bef604c1bc1]) successfully deleted
I20240501 14:09:35.720521 1634 raft_consensus.cc:2229] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:35.720702 1634 raft_consensus.cc:2258] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.721163 1634 ts_tablet_manager.cc:1875] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.723017 1634 ts_tablet_manager.cc:1888] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:35.723121 1634 log.cc:1196] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/92cac7a9b26444e3916d4d7e5dd0032b
I20240501 14:09:35.723382 1634 ts_tablet_manager.cc:1909] T 92cac7a9b26444e3916d4d7e5dd0032b P 1d0bb69611af4876a2b3b33c4766e13a: Deleting consensus metadata
I20240501 14:09:35.723954 1634 tablet_replica.cc:330] T 940c8f54339f4a32be2942682a41c685 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:35.723974 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet 92cac7a9b26444e3916d4d7e5dd0032b (table rebalance_test_table_12 [id=b37dbfc47a574f218dd59bef604c1bc1]) successfully deleted
I20240501 14:09:35.724061 1634 raft_consensus.cc:2229] T 940c8f54339f4a32be2942682a41c685 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.724201 1634 raft_consensus.cc:2258] T 940c8f54339f4a32be2942682a41c685 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.724727 1634 ts_tablet_manager.cc:1875] T 940c8f54339f4a32be2942682a41c685 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:35.726348 1634 ts_tablet_manager.cc:1888] T 940c8f54339f4a32be2942682a41c685 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:35.726423 1634 log.cc:1196] T 940c8f54339f4a32be2942682a41c685 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/940c8f54339f4a32be2942682a41c685
I20240501 14:09:35.726835 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet 940c8f54339f4a32be2942682a41c685 (table rebalance_test_table_9 [id=abd0f09508ae4b6699a526d040fd6f96]) successfully deleted
I20240501 14:09:35.736547 29868 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:35.736646 29868 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:35.738680 29868 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 810e1e79d93a4cd9b4f4deef398ee653
I20240501 14:09:35.739279 32402 tablet_service.cc:1514] Processing DeleteTablet for tablet 810e1e79d93a4cd9b4f4deef398ee653 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:35.739329 329 tablet_service.cc:1514] Processing DeleteTablet for tablet 810e1e79d93a4cd9b4f4deef398ee653 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:35.739602 1649 tablet_replica.cc:330] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:35.739763 1649 raft_consensus.cc:2229] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:35.739784 32534 tablet_service.cc:1514] Processing DeleteTablet for tablet 810e1e79d93a4cd9b4f4deef398ee653 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:35.739970 1649 raft_consensus.cc:2258] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.740180 1648 tablet_replica.cc:330] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:35.740052 1650 tablet_replica.cc:330] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:35.740375 1648 raft_consensus.cc:2229] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.740386 1650 raft_consensus.cc:2229] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.740473 1649 ts_tablet_manager.cc:1875] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.740532 1648 raft_consensus.cc:2258] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.740604 1650 raft_consensus.cc:2258] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.741298 1648 ts_tablet_manager.cc:1875] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.741866 1650 ts_tablet_manager.cc:1875] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.742849 1649 ts_tablet_manager.cc:1888] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:35.742945 1649 log.cc:1196] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/810e1e79d93a4cd9b4f4deef398ee653
I20240501 14:09:35.743242 1649 ts_tablet_manager.cc:1909] T 810e1e79d93a4cd9b4f4deef398ee653 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:09:35.743425 1648 ts_tablet_manager.cc:1888] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:35.743510 1648 log.cc:1196] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/810e1e79d93a4cd9b4f4deef398ee653
I20240501 14:09:35.743748 1648 ts_tablet_manager.cc:1909] T 810e1e79d93a4cd9b4f4deef398ee653 P a91029b90bf9415a89a2f70736759ad3: Deleting consensus metadata
I20240501 14:09:35.744561 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 810e1e79d93a4cd9b4f4deef398ee653 (table rebalancer_dynamic_table_new_name [id=dfd3f624f08b4400bb7905c63a103997]) successfully deleted
I20240501 14:09:35.744686 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 810e1e79d93a4cd9b4f4deef398ee653 (table rebalancer_dynamic_table_new_name [id=dfd3f624f08b4400bb7905c63a103997]) successfully deleted
I20240501 14:09:35.746186 1650 ts_tablet_manager.cc:1888] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:35.746244 1650 log.cc:1196] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/810e1e79d93a4cd9b4f4deef398ee653
I20240501 14:09:35.746412 1650 ts_tablet_manager.cc:1909] T 810e1e79d93a4cd9b4f4deef398ee653 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:35.747170 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet 810e1e79d93a4cd9b4f4deef398ee653 (table rebalancer_dynamic_table_new_name [id=dfd3f624f08b4400bb7905c63a103997]) successfully deleted
I20240501 14:09:35.792939 640 consensus_queue.cc:1035] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.795135 1513 consensus_queue.cc:1035] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.805688 32554 raft_consensus.cc:3037] T bf97db677b9d426486f0fc6d751d667c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.806589 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:35.807298 1548 raft_consensus.cc:482] T bf97db677b9d426486f0fc6d751d667c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:35.807359 32287 consensus_queue.cc:1035] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.807382 1548 raft_consensus.cc:504] T bf97db677b9d426486f0fc6d751d667c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:35.807798 1548 leader_election.cc:288] T bf97db677b9d426486f0fc6d751d667c P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:35.808280 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf97db677b9d426486f0fc6d751d667c" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:35.808526 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf97db677b9d426486f0fc6d751d667c" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:35.809547 32489 leader_election.cc:302] T bf97db677b9d426486f0fc6d751d667c P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3; no voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7
I20240501 14:09:35.809695 1548 raft_consensus.cc:2726] T bf97db677b9d426486f0fc6d751d667c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:09:35.812434 29993 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:35.814242 1138 consensus_queue.cc:1035] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.819437 32534 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:35.823863 32402 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:35.824512 329 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:35.831928 30125 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:35.833544 30386 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:35.841518 29867 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:35.847705 329 tablet_service.cc:1467] Processing CreateTablet for tablet 0478b106598a4f26bb142008cee8c0dc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=895de6eb6fb6443c9f92f97d1a1eba6f]), partition=
I20240501 14:09:35.847859 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 0478b106598a4f26bb142008cee8c0dc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=895de6eb6fb6443c9f92f97d1a1eba6f]), partition=
I20240501 14:09:35.847913 32402 tablet_service.cc:1467] Processing CreateTablet for tablet 0478b106598a4f26bb142008cee8c0dc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=895de6eb6fb6443c9f92f97d1a1eba6f]), partition=
I20240501 14:09:35.848001 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0478b106598a4f26bb142008cee8c0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.848100 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0478b106598a4f26bb142008cee8c0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.848141 32402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0478b106598a4f26bb142008cee8c0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.850322 1465 tablet_bootstrap.cc:492] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:35.850574 1466 tablet_bootstrap.cc:492] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:35.850621 1464 tablet_bootstrap.cc:492] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:35.851199 1465 tablet_bootstrap.cc:654] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.851460 1464 tablet_bootstrap.cc:654] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.851555 1466 tablet_bootstrap.cc:654] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.853083 1464 tablet_bootstrap.cc:492] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:35.853163 1464 ts_tablet_manager.cc:1367] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:35.853610 1464 raft_consensus.cc:348] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:35.853708 1464 raft_consensus.cc:374] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.853739 1464 raft_consensus.cc:729] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.853853 1464 consensus_queue.cc:260] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:35.854154 1464 ts_tablet_manager.cc:1398] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.855020 32665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:35.859403 1465 tablet_bootstrap.cc:492] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:35.859479 1465 ts_tablet_manager.cc:1367] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240501 14:09:35.859892 1465 raft_consensus.cc:348] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:35.859995 1465 raft_consensus.cc:374] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.860023 1465 raft_consensus.cc:729] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.860133 1465 consensus_queue.cc:260] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:35.860404 1465 ts_tablet_manager.cc:1398] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.882872 1466 tablet_bootstrap.cc:492] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:35.882959 1466 ts_tablet_manager.cc:1367] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.001s
I20240501 14:09:35.883363 1466 raft_consensus.cc:348] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:35.883473 1466 raft_consensus.cc:374] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.883498 1466 raft_consensus.cc:729] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.883607 1466 consensus_queue.cc:260] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:35.883836 1466 ts_tablet_manager.cc:1398] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.901878 30014 consensus_queue.cc:237] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:35.903018 30406 raft_consensus.cc:1261] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.903015 348 raft_consensus.cc:1261] T af39c304693645269a6af59190e95cad P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.903350 1513 consensus_queue.cc:1035] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.903509 1689 consensus_queue.cc:1035] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.905090 1689 raft_consensus.cc:2932] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:35.905517 348 raft_consensus.cc:2932] T af39c304693645269a6af59190e95cad P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:35.905465 30406 raft_consensus.cc:2932] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:35.907876 29867 catalog_manager.cc:5484] T af39c304693645269a6af59190e95cad P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:35.910822 30256 tablet_service.cc:1514] Processing DeleteTablet for tablet af39c304693645269a6af59190e95cad with delete_type TABLET_DATA_TOMBSTONED (TS 1d0bb69611af4876a2b3b33c4766e13a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:35.911059 1634 tablet_replica.cc:330] T af39c304693645269a6af59190e95cad P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:35.911193 1634 raft_consensus.cc:2229] T af39c304693645269a6af59190e95cad P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.911350 1634 raft_consensus.cc:2258] T af39c304693645269a6af59190e95cad P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.911805 1634 ts_tablet_manager.cc:1875] T af39c304693645269a6af59190e95cad P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:35.914033 1634 ts_tablet_manager.cc:1888] T af39c304693645269a6af59190e95cad P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:35.914110 1634 log.cc:1196] T af39c304693645269a6af59190e95cad P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/af39c304693645269a6af59190e95cad
I20240501 14:09:35.914614 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet af39c304693645269a6af59190e95cad (table rebalance_test_table_3 [id=b7b3f5bc4b6649c5a97ec9f02da4e967]) successfully deleted
I20240501 14:09:35.957499 1253 consensus_queue.cc:1035] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.961690 562 consensus_queue.cc:1035] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.057417 562 raft_consensus.cc:482] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:36.057533 562 raft_consensus.cc:504] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:36.057905 562 leader_election.cc:288] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:36.058293 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0478b106598a4f26bb142008cee8c0dc" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:36.058302 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0478b106598a4f26bb142008cee8c0dc" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:36.058461 348 raft_consensus.cc:2454] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:36.058490 32554 raft_consensus.cc:2454] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:36.058749 32358 leader_election.cc:302] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:36.058907 562 raft_consensus.cc:2781] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:36.058974 562 raft_consensus.cc:482] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:36.059026 562 raft_consensus.cc:3037] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.059908 562 raft_consensus.cc:504] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:36.060220 562 leader_election.cc:288] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:36.060467 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0478b106598a4f26bb142008cee8c0dc" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:36.060549 348 raft_consensus.cc:3037] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.060541 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0478b106598a4f26bb142008cee8c0dc" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:36.060657 32554 raft_consensus.cc:3037] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.061389 348 raft_consensus.cc:2454] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:36.061394 32554 raft_consensus.cc:2454] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:36.061627 32358 leader_election.cc:302] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:36.061774 562 raft_consensus.cc:2781] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:36.061852 562 raft_consensus.cc:686] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:36.061972 562 consensus_queue.cc:237] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:36.063519 29864 catalog_manager.cc:5484] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:36.077140 29864 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:36.085963 32402 tablet_service.cc:1467] Processing CreateTablet for tablet 588dd26e1820409a9dfccd8ea6a30f49 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=8ff8f244c547410f853c771c11ecdd1e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.085994 30125 tablet_service.cc:1467] Processing CreateTablet for tablet bbff41345a424fce9e328b199d4baa59 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=8ff8f244c547410f853c771c11ecdd1e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.086230 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbff41345a424fce9e328b199d4baa59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.086234 32402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 588dd26e1820409a9dfccd8ea6a30f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.086163 32663 tablet_service.cc:1467] Processing CreateTablet for tablet c7df6b8d76714eebbeacc85f403d3405 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=8ff8f244c547410f853c771c11ecdd1e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.086208 30124 tablet_service.cc:1467] Processing CreateTablet for tablet c7df6b8d76714eebbeacc85f403d3405 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=8ff8f244c547410f853c771c11ecdd1e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.086416 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7df6b8d76714eebbeacc85f403d3405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.086424 32663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7df6b8d76714eebbeacc85f403d3405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.086653 32665 tablet_service.cc:1467] Processing CreateTablet for tablet bbff41345a424fce9e328b199d4baa59 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=8ff8f244c547410f853c771c11ecdd1e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.086850 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbff41345a424fce9e328b199d4baa59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.088455 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 588dd26e1820409a9dfccd8ea6a30f49 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=8ff8f244c547410f853c771c11ecdd1e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.088486 32400 tablet_service.cc:1467] Processing CreateTablet for tablet c7df6b8d76714eebbeacc85f403d3405 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=8ff8f244c547410f853c771c11ecdd1e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.088660 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7df6b8d76714eebbeacc85f403d3405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.088660 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 588dd26e1820409a9dfccd8ea6a30f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.089742 1464 tablet_bootstrap.cc:492] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:36.090698 1464 tablet_bootstrap.cc:654] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.090689 30386 tablet_service.cc:1467] Processing CreateTablet for tablet 588dd26e1820409a9dfccd8ea6a30f49 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=8ff8f244c547410f853c771c11ecdd1e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.090970 30386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 588dd26e1820409a9dfccd8ea6a30f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.091181 1771 tablet_bootstrap.cc:492] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:36.091495 329 tablet_service.cc:1467] Processing CreateTablet for tablet bbff41345a424fce9e328b199d4baa59 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=8ff8f244c547410f853c771c11ecdd1e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.091769 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbff41345a424fce9e328b199d4baa59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.092156 1771 tablet_bootstrap.cc:654] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.093228 1771 tablet_bootstrap.cc:492] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:36.093314 1771 ts_tablet_manager.cc:1367] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:36.093778 1771 raft_consensus.cc:348] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:36.093878 1771 raft_consensus.cc:374] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.093899 1466 tablet_bootstrap.cc:492] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:36.093911 1771 raft_consensus.cc:729] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.094017 1771 consensus_queue.cc:260] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:36.094249 32534 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:36.094286 1771 ts_tablet_manager.cc:1398] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.094818 1466 tablet_bootstrap.cc:654] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.096685 1772 tablet_bootstrap.cc:492] T c7df6b8d76714eebbeacc85f403d3405 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:36.096803 1466 tablet_bootstrap.cc:492] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:36.096889 1466 ts_tablet_manager.cc:1367] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:09:36.097317 1466 raft_consensus.cc:348] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:36.097401 1466 raft_consensus.cc:374] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.097436 1466 raft_consensus.cc:729] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.097546 1772 tablet_bootstrap.cc:654] T c7df6b8d76714eebbeacc85f403d3405 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.097579 1466 consensus_queue.cc:260] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:36.097838 1466 ts_tablet_manager.cc:1398] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.099792 1771 tablet_bootstrap.cc:492] T bbff41345a424fce9e328b199d4baa59 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:36.101044 1771 tablet_bootstrap.cc:654] T bbff41345a424fce9e328b199d4baa59 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.102448 30386 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:36.106889 1775 tablet_bootstrap.cc:492] T 588dd26e1820409a9dfccd8ea6a30f49 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:36.107707 32400 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:36.108837 1775 tablet_bootstrap.cc:654] T 588dd26e1820409a9dfccd8ea6a30f49 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.109236 329 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:36.111117 1772 tablet_bootstrap.cc:492] T c7df6b8d76714eebbeacc85f403d3405 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:36.111248 1772 ts_tablet_manager.cc:1367] T c7df6b8d76714eebbeacc85f403d3405 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 14:09:36.111766 1772 raft_consensus.cc:348] T c7df6b8d76714eebbeacc85f403d3405 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:36.111877 1771 tablet_bootstrap.cc:492] T bbff41345a424fce9e328b199d4baa59 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:36.111918 1772 raft_consensus.cc:374] T c7df6b8d76714eebbeacc85f403d3405 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.111963 1772 raft_consensus.cc:729] T c7df6b8d76714eebbeacc85f403d3405 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.111970 1771 ts_tablet_manager.cc:1367] T bbff41345a424fce9e328b199d4baa59 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:36.112149 1772 consensus_queue.cc:260] T c7df6b8d76714eebbeacc85f403d3405 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:36.112449 1771 raft_consensus.cc:348] T bbff41345a424fce9e328b199d4baa59 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:36.112546 1771 raft_consensus.cc:374] T bbff41345a424fce9e328b199d4baa59 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.112569 1772 ts_tablet_manager.cc:1398] T c7df6b8d76714eebbeacc85f403d3405 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.112578 1771 raft_consensus.cc:729] T bbff41345a424fce9e328b199d4baa59 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.112725 1772 tablet_bootstrap.cc:492] T 588dd26e1820409a9dfccd8ea6a30f49 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:36.112695 1771 consensus_queue.cc:260] T bbff41345a424fce9e328b199d4baa59 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:36.112993 1771 ts_tablet_manager.cc:1398] T bbff41345a424fce9e328b199d4baa59 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.113636 1772 tablet_bootstrap.cc:654] T 588dd26e1820409a9dfccd8ea6a30f49 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.113947 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:36.114627 1775 tablet_bootstrap.cc:492] T 588dd26e1820409a9dfccd8ea6a30f49 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:36.114710 1775 ts_tablet_manager.cc:1367] T 588dd26e1820409a9dfccd8ea6a30f49 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:36.115115 1775 raft_consensus.cc:348] T 588dd26e1820409a9dfccd8ea6a30f49 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:36.115216 1775 raft_consensus.cc:374] T 588dd26e1820409a9dfccd8ea6a30f49 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.115242 1775 raft_consensus.cc:729] T 588dd26e1820409a9dfccd8ea6a30f49 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.115334 1775 consensus_queue.cc:260] T 588dd26e1820409a9dfccd8ea6a30f49 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:36.115586 1775 ts_tablet_manager.cc:1398] T 588dd26e1820409a9dfccd8ea6a30f49 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:36.117453 32665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:36.121789 29993 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:36.124548 1464 tablet_bootstrap.cc:492] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:36.124647 1464 ts_tablet_manager.cc:1367] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20240501 14:09:36.125066 1464 raft_consensus.cc:348] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:36.125175 1464 raft_consensus.cc:374] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.125211 1464 raft_consensus.cc:729] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.125330 1464 consensus_queue.cc:260] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:36.125612 1464 ts_tablet_manager.cc:1398] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.125759 1464 tablet_bootstrap.cc:492] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:36.127265 1464 tablet_bootstrap.cc:654] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.130542 30125 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:36.136646 1772 tablet_bootstrap.cc:492] T 588dd26e1820409a9dfccd8ea6a30f49 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:36.136827 1772 ts_tablet_manager.cc:1367] T 588dd26e1820409a9dfccd8ea6a30f49 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240501 14:09:36.137250 1772 raft_consensus.cc:348] T 588dd26e1820409a9dfccd8ea6a30f49 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:36.137415 1772 raft_consensus.cc:374] T 588dd26e1820409a9dfccd8ea6a30f49 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.137478 1772 raft_consensus.cc:729] T 588dd26e1820409a9dfccd8ea6a30f49 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.137657 1772 consensus_queue.cc:260] T 588dd26e1820409a9dfccd8ea6a30f49 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:36.137929 1138 raft_consensus.cc:482] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:36.138018 1138 raft_consensus.cc:504] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:36.138456 1138 leader_election.cc:288] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:36.139197 1772 ts_tablet_manager.cc:1398] T 588dd26e1820409a9dfccd8ea6a30f49 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:36.139422 1772 tablet_bootstrap.cc:492] T bbff41345a424fce9e328b199d4baa59 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:36.139957 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbff41345a424fce9e328b199d4baa59" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:36.140574 1772 tablet_bootstrap.cc:654] T bbff41345a424fce9e328b199d4baa59 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
W20240501 14:09:36.140503 32750 leader_election.cc:341] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:36.147563 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbff41345a424fce9e328b199d4baa59" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:36.147682 30146 raft_consensus.cc:2454] T bbff41345a424fce9e328b199d4baa59 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:36.147907 32753 leader_election.cc:302] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, d35c16dcb506426993811bb2519d7db3; no voters: db935501e9104de5b15c8780681d6c38
I20240501 14:09:36.148195 1138 raft_consensus.cc:2781] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:36.148269 1138 raft_consensus.cc:482] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:36.148315 1138 raft_consensus.cc:3037] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.149298 1138 raft_consensus.cc:504] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:36.149634 1138 leader_election.cc:288] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:36.149922 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbff41345a424fce9e328b199d4baa59" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
W20240501 14:09:36.150280 32750 leader_election.cc:341] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:36.152699 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbff41345a424fce9e328b199d4baa59" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:36.152815 30146 raft_consensus.cc:3037] T bbff41345a424fce9e328b199d4baa59 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.153733 30146 raft_consensus.cc:2454] T bbff41345a424fce9e328b199d4baa59 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:36.154058 32753 leader_election.cc:302] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, d35c16dcb506426993811bb2519d7db3; no voters: db935501e9104de5b15c8780681d6c38
I20240501 14:09:36.154695 1138 raft_consensus.cc:2781] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:36.154778 1138 raft_consensus.cc:686] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:36.154911 1138 consensus_queue.cc:237] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:36.157184 29868 catalog_manager.cc:5484] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:36.165120 1772 tablet_bootstrap.cc:492] T bbff41345a424fce9e328b199d4baa59 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:36.165205 1772 ts_tablet_manager.cc:1367] T bbff41345a424fce9e328b199d4baa59 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.000s
I20240501 14:09:36.165557 1772 raft_consensus.cc:348] T bbff41345a424fce9e328b199d4baa59 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:36.165640 1772 raft_consensus.cc:374] T bbff41345a424fce9e328b199d4baa59 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.165671 1772 raft_consensus.cc:729] T bbff41345a424fce9e328b199d4baa59 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.165771 1772 consensus_queue.cc:260] T bbff41345a424fce9e328b199d4baa59 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:36.166040 1772 ts_tablet_manager.cc:1398] T bbff41345a424fce9e328b199d4baa59 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.167796 1076 raft_consensus.cc:482] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:36.167876 1076 raft_consensus.cc:504] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:36.168181 1076 leader_election.cc:288] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:36.169051 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7df6b8d76714eebbeacc85f403d3405" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:36.169170 32683 raft_consensus.cc:2454] T c7df6b8d76714eebbeacc85f403d3405 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:36.169589 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7df6b8d76714eebbeacc85f403d3405" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:36.169818 30079 leader_election.cc:302] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d35c16dcb506426993811bb2519d7db3, db935501e9104de5b15c8780681d6c38; no voters:
W20240501 14:09:36.170112 30081 leader_election.cc:341] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:36.170496 500 raft_consensus.cc:2781] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:36.170543 500 raft_consensus.cc:482] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:36.170576 500 raft_consensus.cc:3037] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.171497 500 raft_consensus.cc:504] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:36.171774 500 leader_election.cc:288] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:36.171826 1464 tablet_bootstrap.cc:492] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:36.171906 1464 ts_tablet_manager.cc:1367] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.000s
I20240501 14:09:36.172225 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7df6b8d76714eebbeacc85f403d3405" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:36.172345 32683 raft_consensus.cc:3037] T c7df6b8d76714eebbeacc85f403d3405 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.172399 1464 raft_consensus.cc:348] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:36.172500 1464 raft_consensus.cc:374] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.172555 1464 raft_consensus.cc:729] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.172667 1464 consensus_queue.cc:260] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:36.172920 1464 ts_tablet_manager.cc:1398] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.173174 32683 raft_consensus.cc:2454] T c7df6b8d76714eebbeacc85f403d3405 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:36.174404 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7df6b8d76714eebbeacc85f403d3405" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:36.174510 32416 raft_consensus.cc:3037] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.175359 32416 raft_consensus.cc:2454] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:36.175750 30079 leader_election.cc:302] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d35c16dcb506426993811bb2519d7db3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:36.177599 1076 raft_consensus.cc:2781] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:36.177676 1076 raft_consensus.cc:686] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:36.177783 1076 consensus_queue.cc:237] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:36.179811 29868 catalog_manager.cc:5484] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:36.182032 562 raft_consensus.cc:482] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:36.182113 562 raft_consensus.cc:504] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:36.182534 562 leader_election.cc:288] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:36.183383 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588dd26e1820409a9dfccd8ea6a30f49" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:36.183483 32683 raft_consensus.cc:2454] T 588dd26e1820409a9dfccd8ea6a30f49 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:36.183743 32356 leader_election.cc:302] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:36.183851 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588dd26e1820409a9dfccd8ea6a30f49" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:36.183903 562 raft_consensus.cc:2781] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:36.183967 562 raft_consensus.cc:482] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:36.183971 30406 raft_consensus.cc:2454] T 588dd26e1820409a9dfccd8ea6a30f49 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:36.184005 562 raft_consensus.cc:3037] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.184907 562 raft_consensus.cc:504] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:36.185225 562 leader_election.cc:288] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:36.185659 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588dd26e1820409a9dfccd8ea6a30f49" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:36.185788 30406 raft_consensus.cc:3037] T 588dd26e1820409a9dfccd8ea6a30f49 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.185819 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588dd26e1820409a9dfccd8ea6a30f49" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:36.185904 32683 raft_consensus.cc:3037] T 588dd26e1820409a9dfccd8ea6a30f49 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.186827 30406 raft_consensus.cc:2454] T 588dd26e1820409a9dfccd8ea6a30f49 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:36.186918 32683 raft_consensus.cc:2454] T 588dd26e1820409a9dfccd8ea6a30f49 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:36.187209 32356 leader_election.cc:302] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:36.187708 562 raft_consensus.cc:2781] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:36.187798 562 raft_consensus.cc:686] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:36.187914 562 consensus_queue.cc:237] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:36.189561 29868 catalog_manager.cc:5484] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:36.192294 29868 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:36.197021 348 raft_consensus.cc:1261] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:36.197355 32554 raft_consensus.cc:1261] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:36.197392 1253 consensus_queue.cc:1035] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.197788 1253 consensus_queue.cc:1035] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.199877 29868 catalog_manager.cc:5576] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497) does not have the latest schema for tablet 0478b106598a4f26bb142008cee8c0dc (table rebalancer_dynamic_table [id=895de6eb6fb6443c9f92f97d1a1eba6f]). Expected version 1 got 0
I20240501 14:09:36.200183 29864 catalog_manager.cc:5576] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937) does not have the latest schema for tablet 0478b106598a4f26bb142008cee8c0dc (table rebalancer_dynamic_table [id=895de6eb6fb6443c9f92f97d1a1eba6f]). Expected version 1 got 0
I20240501 14:09:36.201758 1799 tablet.cc:1722] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:36.201828 29864 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 14:09:36.201915 29864 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 14:09:36.202064 1799 tablet.cc:1620] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:36.202453 1798 tablet.cc:1722] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:36.203094 29873 catalog_manager.cc:5576] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937) does not have the latest schema for tablet 0478b106598a4f26bb142008cee8c0dc (table rebalancer_dynamic_table [id=895de6eb6fb6443c9f92f97d1a1eba6f]). Expected version 1 got 0
I20240501 14:09:36.203866 29864 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 4 replicas of tablet 5da2beba68ba414281025fcee91ca0d0
I20240501 14:09:36.204149 29864 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 855d8906d14e46ebacfd866861d287e7
I20240501 14:09:36.204339 29864 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 8a7582b1c4c040849ad1507ccbaf0584
I20240501 14:09:36.204545 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 5da2beba68ba414281025fcee91ca0d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:36.204716 543 tablet_replica.cc:330] T 5da2beba68ba414281025fcee91ca0d0 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:36.204747 32534 tablet_service.cc:1514] Processing DeleteTablet for tablet 5da2beba68ba414281025fcee91ca0d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:36.204852 543 raft_consensus.cc:2229] T 5da2beba68ba414281025fcee91ca0d0 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.204903 1650 tablet_replica.cc:330] T 5da2beba68ba414281025fcee91ca0d0 P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:36.204994 543 raft_consensus.cc:2258] T 5da2beba68ba414281025fcee91ca0d0 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.205052 1650 raft_consensus.cc:2229] T 5da2beba68ba414281025fcee91ca0d0 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.205113 29994 tablet_service.cc:1514] Processing DeleteTablet for tablet 855d8906d14e46ebacfd866861d287e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:36.205149 1650 raft_consensus.cc:2258] T 5da2beba68ba414281025fcee91ca0d0 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.205303 30256 tablet_service.cc:1514] Processing DeleteTablet for tablet 855d8906d14e46ebacfd866861d287e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:36.205461 1634 tablet_replica.cc:330] T 855d8906d14e46ebacfd866861d287e7 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:36.205480 543 ts_tablet_manager.cc:1875] T 5da2beba68ba414281025fcee91ca0d0 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.205578 1634 raft_consensus.cc:2229] T 855d8906d14e46ebacfd866861d287e7 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.205686 1634 raft_consensus.cc:2258] T 855d8906d14e46ebacfd866861d287e7 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.205729 1650 ts_tablet_manager.cc:1875] T 5da2beba68ba414281025fcee91ca0d0 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.205991 30387 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a7582b1c4c040849ad1507ccbaf0584 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:36.206144 1634 ts_tablet_manager.cc:1875] T 855d8906d14e46ebacfd866861d287e7 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.206154 546 tablet_replica.cc:330] T 8a7582b1c4c040849ad1507ccbaf0584 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:36.206272 546 raft_consensus.cc:2229] T 8a7582b1c4c040849ad1507ccbaf0584 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.206372 546 raft_consensus.cc:2258] T 8a7582b1c4c040849ad1507ccbaf0584 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.206728 30386 tablet_service.cc:1514] Processing DeleteTablet for tablet 5da2beba68ba414281025fcee91ca0d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:36.206844 546 ts_tablet_manager.cc:1875] T 8a7582b1c4c040849ad1507ccbaf0584 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.207155 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a7582b1c4c040849ad1507ccbaf0584 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:36.207337 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet 5da2beba68ba414281025fcee91ca0d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:36.207463 29995 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a7582b1c4c040849ad1507ccbaf0584 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:36.207568 545 tablet_replica.cc:330] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:36.207675 545 raft_consensus.cc:2229] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:36.207880 545 raft_consensus.cc:2258] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.208290 1634 ts_tablet_manager.cc:1888] T 855d8906d14e46ebacfd866861d287e7 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:36.208355 1634 log.cc:1196] T 855d8906d14e46ebacfd866861d287e7 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/855d8906d14e46ebacfd866861d287e7
I20240501 14:09:36.208355 543 ts_tablet_manager.cc:1888] T 5da2beba68ba414281025fcee91ca0d0 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:36.208410 543 log.cc:1196] T 5da2beba68ba414281025fcee91ca0d0 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/5da2beba68ba414281025fcee91ca0d0
I20240501 14:09:36.208408 545 ts_tablet_manager.cc:1875] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.208686 543 ts_tablet_manager.cc:1909] T 5da2beba68ba414281025fcee91ca0d0 P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:36.208781 1634 ts_tablet_manager.cc:1909] T 855d8906d14e46ebacfd866861d287e7 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting consensus metadata
I20240501 14:09:36.209067 546 ts_tablet_manager.cc:1888] T 8a7582b1c4c040849ad1507ccbaf0584 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:36.209124 546 log.cc:1196] T 8a7582b1c4c040849ad1507ccbaf0584 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/8a7582b1c4c040849ad1507ccbaf0584
I20240501 14:09:36.206845 30388 tablet_service.cc:1514] Processing DeleteTablet for tablet 855d8906d14e46ebacfd866861d287e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:36.209733 543 tablet_replica.cc:330] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:36.209815 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 5da2beba68ba414281025fcee91ca0d0 (table rebalance_test_table_14 [id=a61705ba694e47f3818ac35bcffaee89]) successfully deleted
I20240501 14:09:36.209856 543 raft_consensus.cc:2229] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:36.210038 543 raft_consensus.cc:2258] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.210017 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet 855d8906d14e46ebacfd866861d287e7 (table rebalance_test_table_14 [id=a61705ba694e47f3818ac35bcffaee89]) successfully deleted
I20240501 14:09:36.210479 543 ts_tablet_manager.cc:1875] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.210660 545 ts_tablet_manager.cc:1888] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:36.210734 545 log.cc:1196] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/855d8906d14e46ebacfd866861d287e7
I20240501 14:09:36.210937 546 ts_tablet_manager.cc:1909] T 8a7582b1c4c040849ad1507ccbaf0584 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:36.210999 545 ts_tablet_manager.cc:1909] T 855d8906d14e46ebacfd866861d287e7 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:36.211509 545 tablet_replica.cc:330] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:36.211633 545 raft_consensus.cc:2229] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:36.211652 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 8a7582b1c4c040849ad1507ccbaf0584 (table rebalance_test_table_14 [id=a61705ba694e47f3818ac35bcffaee89]) successfully deleted
I20240501 14:09:36.211784 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 855d8906d14e46ebacfd866861d287e7 (table rebalance_test_table_14 [id=a61705ba694e47f3818ac35bcffaee89]) successfully deleted
I20240501 14:09:36.211828 545 raft_consensus.cc:2258] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.212608 546 tablet_replica.cc:330] T 5da2beba68ba414281025fcee91ca0d0 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:36.212728 546 raft_consensus.cc:2229] T 5da2beba68ba414281025fcee91ca0d0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.213434 543 ts_tablet_manager.cc:1888] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:36.213506 543 log.cc:1196] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/8a7582b1c4c040849ad1507ccbaf0584
I20240501 14:09:36.212852 546 raft_consensus.cc:2258] T 5da2beba68ba414281025fcee91ca0d0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.213763 543 ts_tablet_manager.cc:1909] T 8a7582b1c4c040849ad1507ccbaf0584 P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:36.213910 1650 ts_tablet_manager.cc:1888] T 5da2beba68ba414281025fcee91ca0d0 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:36.213986 1650 log.cc:1196] T 5da2beba68ba414281025fcee91ca0d0 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/5da2beba68ba414281025fcee91ca0d0
I20240501 14:09:36.214269 1650 ts_tablet_manager.cc:1909] T 5da2beba68ba414281025fcee91ca0d0 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:36.214304 546 ts_tablet_manager.cc:1875] T 5da2beba68ba414281025fcee91ca0d0 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.214365 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 8a7582b1c4c040849ad1507ccbaf0584 (table rebalance_test_table_14 [id=a61705ba694e47f3818ac35bcffaee89]) successfully deleted
I20240501 14:09:36.215423 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet 5da2beba68ba414281025fcee91ca0d0 (table rebalance_test_table_14 [id=a61705ba694e47f3818ac35bcffaee89]) successfully deleted
I20240501 14:09:36.215543 545 ts_tablet_manager.cc:1875] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.216279 1798 tablet.cc:1620] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:36.216403 30146 consensus_queue.cc:237] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:36.217759 32416 raft_consensus.cc:1261] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:36.217877 30264 raft_consensus.cc:1261] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:36.218228 500 consensus_queue.cc:1035] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:36.218587 500 consensus_queue.cc:1035] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:36.218763 29864 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=895de6eb6fb6443c9f92f97d1a1eba6f] alter complete (version 1)
I20240501 14:09:36.220672 545 ts_tablet_manager.cc:1888] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:36.220734 546 ts_tablet_manager.cc:1888] T 5da2beba68ba414281025fcee91ca0d0 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:36.220803 545 log.cc:1196] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/5da2beba68ba414281025fcee91ca0d0
I20240501 14:09:36.220854 546 log.cc:1196] T 5da2beba68ba414281025fcee91ca0d0 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/5da2beba68ba414281025fcee91ca0d0
I20240501 14:09:36.221127 545 ts_tablet_manager.cc:1909] T 5da2beba68ba414281025fcee91ca0d0 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:36.221127 546 ts_tablet_manager.cc:1909] T 5da2beba68ba414281025fcee91ca0d0 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:36.221699 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 5da2beba68ba414281025fcee91ca0d0 (table rebalance_test_table_14 [id=a61705ba694e47f3818ac35bcffaee89]) successfully deleted
I20240501 14:09:36.221771 545 tablet_replica.cc:330] T 8a7582b1c4c040849ad1507ccbaf0584 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:36.221895 545 raft_consensus.cc:2229] T 8a7582b1c4c040849ad1507ccbaf0584 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.221932 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 5da2beba68ba414281025fcee91ca0d0 (table rebalance_test_table_14 [id=a61705ba694e47f3818ac35bcffaee89]) successfully deleted
I20240501 14:09:36.222034 545 raft_consensus.cc:2258] T 8a7582b1c4c040849ad1507ccbaf0584 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.222131 546 tablet_replica.cc:330] T 855d8906d14e46ebacfd866861d287e7 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:36.222256 546 raft_consensus.cc:2229] T 855d8906d14e46ebacfd866861d287e7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.222358 546 raft_consensus.cc:2258] T 855d8906d14e46ebacfd866861d287e7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.222496 545 ts_tablet_manager.cc:1875] T 8a7582b1c4c040849ad1507ccbaf0584 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.222750 546 ts_tablet_manager.cc:1875] T 855d8906d14e46ebacfd866861d287e7 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.224501 546 ts_tablet_manager.cc:1888] T 855d8906d14e46ebacfd866861d287e7 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:36.224567 546 log.cc:1196] T 855d8906d14e46ebacfd866861d287e7 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/855d8906d14e46ebacfd866861d287e7
I20240501 14:09:36.224776 546 ts_tablet_manager.cc:1909] T 855d8906d14e46ebacfd866861d287e7 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:36.225358 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 855d8906d14e46ebacfd866861d287e7 (table rebalance_test_table_14 [id=a61705ba694e47f3818ac35bcffaee89]) successfully deleted
I20240501 14:09:36.226411 1076 raft_consensus.cc:2932] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:36.226954 32416 raft_consensus.cc:2932] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:36.228333 545 ts_tablet_manager.cc:1888] T 8a7582b1c4c040849ad1507ccbaf0584 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:36.228569 545 log.cc:1196] T 8a7582b1c4c040849ad1507ccbaf0584 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/8a7582b1c4c040849ad1507ccbaf0584
I20240501 14:09:36.228884 545 ts_tablet_manager.cc:1909] T 8a7582b1c4c040849ad1507ccbaf0584 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:36.229192 30264 raft_consensus.cc:2932] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:36.229776 29873 catalog_manager.cc:5484] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index 4 to 5, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:36.231222 29868 catalog_manager.cc:5576] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497) does not have the latest schema for tablet 0478b106598a4f26bb142008cee8c0dc (table rebalancer_dynamic_table [id=895de6eb6fb6443c9f92f97d1a1eba6f]). Expected version 1 got 0
I20240501 14:09:36.232440 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 8a7582b1c4c040849ad1507ccbaf0584 (table rebalance_test_table_14 [id=a61705ba694e47f3818ac35bcffaee89]) successfully deleted
I20240501 14:09:36.232764 29995 tablet_service.cc:1514] Processing DeleteTablet for tablet 796cef2485c94b76a5f3e1944b73dcf6 with delete_type TABLET_DATA_TOMBSTONED (TS 93f2518fbadf4f9991cda93afacb363e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:36.233073 545 tablet_replica.cc:330] T 796cef2485c94b76a5f3e1944b73dcf6 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:36.233243 545 raft_consensus.cc:2229] T 796cef2485c94b76a5f3e1944b73dcf6 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.233438 545 raft_consensus.cc:2258] T 796cef2485c94b76a5f3e1944b73dcf6 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.233886 545 ts_tablet_manager.cc:1875] T 796cef2485c94b76a5f3e1944b73dcf6 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:36.235736 545 ts_tablet_manager.cc:1888] T 796cef2485c94b76a5f3e1944b73dcf6 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:36.235846 545 log.cc:1196] T 796cef2485c94b76a5f3e1944b73dcf6 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/796cef2485c94b76a5f3e1944b73dcf6
I20240501 14:09:36.236428 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 796cef2485c94b76a5f3e1944b73dcf6 (table rebalance_test_table_16 [id=86d7db413f934f379c4369701c5c13cd]) successfully deleted
I20240501 14:09:36.237365 1808 tablet.cc:1722] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:36.237672 1808 tablet.cc:1620] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:36.313185 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:36.315692 32534 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:36.316320 30388 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:36.320963 969 raft_consensus.cc:482] T bbff41345a424fce9e328b199d4baa59 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:36.321061 969 raft_consensus.cc:504] T bbff41345a424fce9e328b199d4baa59 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:36.321213 29867 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:36.321540 969 leader_election.cc:288] T bbff41345a424fce9e328b199d4baa59 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:36.321955 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbff41345a424fce9e328b199d4baa59" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:36.323639 32665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:36.323838 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbff41345a424fce9e328b199d4baa59" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:36.324039 30146 raft_consensus.cc:2379] T bbff41345a424fce9e328b199d4baa59 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db935501e9104de5b15c8780681d6c38 in current term 1: Already voted for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in this term.
I20240501 14:09:36.325636 329 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:36.325860 29995 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:36.325834 32621 leader_election.cc:302] T bbff41345a424fce9e328b199d4baa59 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db935501e9104de5b15c8780681d6c38; no voters: 9ecd907cf5634ac98f3f54f3dbec6849, d35c16dcb506426993811bb2519d7db3
I20240501 14:09:36.326010 969 raft_consensus.cc:3037] T bbff41345a424fce9e328b199d4baa59 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.327073 969 raft_consensus.cc:2726] T bbff41345a424fce9e328b199d4baa59 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:09:36.332427 1798 tablet.cc:1722] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:36.332755 1798 tablet.cc:1620] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:36.333324 1799 tablet.cc:1722] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:36.333580 1799 tablet.cc:1620] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:36.335026 1808 tablet.cc:1722] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:36.335309 1808 tablet.cc:1620] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:36.340226 29870 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=895de6eb6fb6443c9f92f97d1a1eba6f] alter complete (version 2)
I20240501 14:09:36.345052 32400 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:36.348126 30125 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:36.397909 30406 consensus_queue.cc:237] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:36.399149 32683 raft_consensus.cc:1261] T 3a04e958f3d049139ff74202611af725 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:36.399174 30264 raft_consensus.cc:1261] T 3a04e958f3d049139ff74202611af725 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:36.399564 995 consensus_queue.cc:1035] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:36.399744 1557 consensus_queue.cc:1035] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:36.401654 1877 raft_consensus.cc:2932] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:36.402104 30264 raft_consensus.cc:2932] T 3a04e958f3d049139ff74202611af725 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:36.402518 32683 raft_consensus.cc:2932] T 3a04e958f3d049139ff74202611af725 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:36.405468 29867 catalog_manager.cc:5484] T 3a04e958f3d049139ff74202611af725 P db935501e9104de5b15c8780681d6c38 reported cstate change: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:36.408378 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a04e958f3d049139ff74202611af725 with delete_type TABLET_DATA_TOMBSTONED (TS d35c16dcb506426993811bb2519d7db3 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:36.408681 543 tablet_replica.cc:330] T 3a04e958f3d049139ff74202611af725 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:36.408836 543 raft_consensus.cc:2229] T 3a04e958f3d049139ff74202611af725 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.408977 543 raft_consensus.cc:2258] T 3a04e958f3d049139ff74202611af725 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.409437 543 ts_tablet_manager.cc:1875] T 3a04e958f3d049139ff74202611af725 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:36.411449 543 ts_tablet_manager.cc:1888] T 3a04e958f3d049139ff74202611af725 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:36.411499 543 log.cc:1196] T 3a04e958f3d049139ff74202611af725 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/3a04e958f3d049139ff74202611af725
I20240501 14:09:36.411921 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 3a04e958f3d049139ff74202611af725 (table rebalance_test_table_5 [id=440878c63b0f4706b34fd42316991270]) successfully deleted
I20240501 14:09:36.444924 29870 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:36.450811 1798 tablet.cc:1722] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:36.451964 1808 tablet.cc:1722] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:36.454741 1799 tablet.cc:1722] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:36.457800 29868 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=895de6eb6fb6443c9f92f97d1a1eba6f] alter complete (version 3)
I20240501 14:09:36.494879 329 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:36.495167 29995 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:36.499636 32665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:36.502364 32400 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:36.502609 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:36.504005 32534 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:36.513199 30125 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:36.513911 30388 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:36.549523 1138 consensus_queue.cc:1035] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.552752 640 consensus_queue.cc:1035] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.561393 29869 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:36.561517 29869 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:36.561801 30264 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3b6a526c2ed84e1780ec392ef39d09b0"
dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51958
I20240501 14:09:36.561944 30264 raft_consensus.cc:595] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:36.563536 29869 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 0478b106598a4f26bb142008cee8c0dc
I20240501 14:09:36.564172 329 tablet_service.cc:1514] Processing DeleteTablet for tablet 0478b106598a4f26bb142008cee8c0dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:36.564323 32400 tablet_service.cc:1514] Processing DeleteTablet for tablet 0478b106598a4f26bb142008cee8c0dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:36.564458 32534 tablet_service.cc:1514] Processing DeleteTablet for tablet 0478b106598a4f26bb142008cee8c0dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:36.564656 1650 tablet_replica.cc:330] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:36.564796 1650 raft_consensus.cc:2229] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.564862 1944 tablet_replica.cc:330] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:36.564851 1943 tablet_replica.cc:330] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:36.564911 1650 raft_consensus.cc:2258] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.564987 1943 raft_consensus.cc:2229] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:36.564987 1944 raft_consensus.cc:2229] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.565158 1944 raft_consensus.cc:2258] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.565250 1943 raft_consensus.cc:2258] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.565711 1650 ts_tablet_manager.cc:1875] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.565744 1944 ts_tablet_manager.cc:1875] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.566112 1943 ts_tablet_manager.cc:1875] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.567879 1650 ts_tablet_manager.cc:1888] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:36.567943 1650 log.cc:1196] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/0478b106598a4f26bb142008cee8c0dc
I20240501 14:09:36.567945 1944 ts_tablet_manager.cc:1888] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:36.568010 1944 log.cc:1196] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/0478b106598a4f26bb142008cee8c0dc
I20240501 14:09:36.568226 1650 ts_tablet_manager.cc:1909] T 0478b106598a4f26bb142008cee8c0dc P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:36.568286 1944 ts_tablet_manager.cc:1909] T 0478b106598a4f26bb142008cee8c0dc P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:09:36.568542 1943 ts_tablet_manager.cc:1888] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:36.568609 1943 log.cc:1196] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/0478b106598a4f26bb142008cee8c0dc
I20240501 14:09:36.568876 1943 ts_tablet_manager.cc:1909] T 0478b106598a4f26bb142008cee8c0dc P a91029b90bf9415a89a2f70736759ad3: Deleting consensus metadata
I20240501 14:09:36.569069 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet 0478b106598a4f26bb142008cee8c0dc (table rebalancer_dynamic_table_new_name [id=895de6eb6fb6443c9f92f97d1a1eba6f]) successfully deleted
I20240501 14:09:36.569595 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 0478b106598a4f26bb142008cee8c0dc (table rebalancer_dynamic_table_new_name [id=895de6eb6fb6443c9f92f97d1a1eba6f]) successfully deleted
I20240501 14:09:36.569749 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 0478b106598a4f26bb142008cee8c0dc (table rebalancer_dynamic_table_new_name [id=895de6eb6fb6443c9f92f97d1a1eba6f]) successfully deleted
I20240501 14:09:36.581528 620 raft_consensus.cc:979] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a: : Instructing follower d35c16dcb506426993811bb2519d7db3 to start an election
I20240501 14:09:36.581630 620 raft_consensus.cc:1067] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Signalling peer d35c16dcb506426993811bb2519d7db3 to start an election
I20240501 14:09:36.582180 30146 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3b6a526c2ed84e1780ec392ef39d09b0"
dest_uuid: "d35c16dcb506426993811bb2519d7db3"
from {username='slave'} at 127.22.252.131:47777
I20240501 14:09:36.582346 30146 raft_consensus.cc:482] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:36.582417 30146 raft_consensus.cc:3037] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:36.583338 30146 raft_consensus.cc:504] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:36.583835 30146 leader_election.cc:288] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:36.584327 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6a526c2ed84e1780ec392ef39d09b0" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:36.584444 30406 raft_consensus.cc:3037] T 3b6a526c2ed84e1780ec392ef39d09b0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:36.584509 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6a526c2ed84e1780ec392ef39d09b0" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:36.584625 30264 raft_consensus.cc:3032] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:36.584687 30264 raft_consensus.cc:729] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:36.584841 30264 consensus_queue.cc:260] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:36.585049 30264 raft_consensus.cc:3037] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:36.585433 30406 raft_consensus.cc:2454] T 3b6a526c2ed84e1780ec392ef39d09b0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 2.
I20240501 14:09:36.585980 30264 raft_consensus.cc:2454] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 2.
I20240501 14:09:36.586192 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6a526c2ed84e1780ec392ef39d09b0" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:36.586285 348 raft_consensus.cc:3037] T 3b6a526c2ed84e1780ec392ef39d09b0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:36.587281 348 raft_consensus.cc:2454] T 3b6a526c2ed84e1780ec392ef39d09b0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 2.
I20240501 14:09:36.587567 30081 leader_election.cc:302] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:36.587780 500 raft_consensus.cc:2781] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:36.587846 500 raft_consensus.cc:686] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [term 2 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:36.587958 500 consensus_queue.cc:237] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:36.589808 29869 catalog_manager.cc:5484] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 1 to 2, leader changed from 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 2 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:36.647462 30256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:36.648648 29995 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:36.649441 329 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:36.651477 30125 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:36.652040 32534 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:36.661370 32400 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:36.662147 30388 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:36.662812 562 consensus_queue.cc:1035] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.669551 1253 consensus_queue.cc:1035] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.673099 29867 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:36.678438 30125 tablet_service.cc:1467] Processing CreateTablet for tablet 52acda15aaec4b6fbab788e8bbe0ff30 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=593b9102cd8c4770a852a8a5ac1db30d]), partition=
I20240501 14:09:36.678508 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 52acda15aaec4b6fbab788e8bbe0ff30 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=593b9102cd8c4770a852a8a5ac1db30d]), partition=
I20240501 14:09:36.678664 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52acda15aaec4b6fbab788e8bbe0ff30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.678725 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52acda15aaec4b6fbab788e8bbe0ff30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.680899 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 52acda15aaec4b6fbab788e8bbe0ff30 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=593b9102cd8c4770a852a8a5ac1db30d]), partition=
I20240501 14:09:36.681175 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52acda15aaec4b6fbab788e8bbe0ff30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.688324 1990 tablet_bootstrap.cc:492] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:36.689988 1988 tablet_bootstrap.cc:492] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:36.690899 1990 tablet_bootstrap.cc:654] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.690771 1988 tablet_bootstrap.cc:654] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.697533 1076 consensus_queue.cc:1035] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.698833 29867 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:36.702395 32665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:36.703001 1989 tablet_bootstrap.cc:492] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:36.703850 1989 tablet_bootstrap.cc:654] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.704599 1990 tablet_bootstrap.cc:492] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:36.704795 1990 ts_tablet_manager.cc:1367] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 14:09:36.704936 1989 tablet_bootstrap.cc:492] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:36.705017 1989 ts_tablet_manager.cc:1367] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:36.705310 1990 raft_consensus.cc:348] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:36.705404 1990 raft_consensus.cc:374] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.705431 1990 raft_consensus.cc:729] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.705478 1989 raft_consensus.cc:348] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:36.705533 1990 consensus_queue.cc:260] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:36.705578 1989 raft_consensus.cc:374] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.705608 1989 raft_consensus.cc:729] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.705711 1989 consensus_queue.cc:260] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:36.705986 1988 tablet_bootstrap.cc:492] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:36.705994 1989 ts_tablet_manager.cc:1398] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:36.706060 1988 ts_tablet_manager.cc:1367] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.002s
I20240501 14:09:36.706457 1988 raft_consensus.cc:348] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:36.706554 1988 raft_consensus.cc:374] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.707353 1990 ts_tablet_manager.cc:1398] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:36.706580 1988 raft_consensus.cc:729] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.707599 1988 consensus_queue.cc:260] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:36.708037 1988 ts_tablet_manager.cc:1398] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:36.711268 29870 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 14:09:36.711377 29870 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 14:09:36.712591 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3a90c03de64d6891dda267167d1d7b (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=c8e44abe11554acda26e71b8a11e2e0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.712855 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3a90c03de64d6891dda267167d1d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.712823 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3a90c03de64d6891dda267167d1d7b (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=c8e44abe11554acda26e71b8a11e2e0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.713070 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3a90c03de64d6891dda267167d1d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.714881 1989 tablet_bootstrap.cc:492] T 7b3a90c03de64d6891dda267167d1d7b P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:36.715893 1989 tablet_bootstrap.cc:654] T 7b3a90c03de64d6891dda267167d1d7b P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.716591 29995 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3a90c03de64d6891dda267167d1d7b (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=c8e44abe11554acda26e71b8a11e2e0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.716845 29995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3a90c03de64d6891dda267167d1d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.718957 29993 tablet_service.cc:1467] Processing CreateTablet for tablet c90d65f39c3540f48c0d85b28130e76a (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=c8e44abe11554acda26e71b8a11e2e0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.719149 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c90d65f39c3540f48c0d85b28130e76a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.720276 1989 tablet_bootstrap.cc:492] T 7b3a90c03de64d6891dda267167d1d7b P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:36.720374 1989 ts_tablet_manager.cc:1367] T 7b3a90c03de64d6891dda267167d1d7b P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 14:09:36.720690 1999 tablet_bootstrap.cc:492] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:36.720767 1989 raft_consensus.cc:348] T 7b3a90c03de64d6891dda267167d1d7b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:36.720865 1989 raft_consensus.cc:374] T 7b3a90c03de64d6891dda267167d1d7b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.720896 1989 raft_consensus.cc:729] T 7b3a90c03de64d6891dda267167d1d7b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.720170 32534 tablet_service.cc:1467] Processing CreateTablet for tablet b26b6d19699047afaee57d6f2c887664 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=c8e44abe11554acda26e71b8a11e2e0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.721004 32400 tablet_service.cc:1467] Processing CreateTablet for tablet c90d65f39c3540f48c0d85b28130e76a (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=c8e44abe11554acda26e71b8a11e2e0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.721164 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26b6d19699047afaee57d6f2c887664. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.721217 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c90d65f39c3540f48c0d85b28130e76a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.721367 32402 tablet_service.cc:1467] Processing CreateTablet for tablet b26b6d19699047afaee57d6f2c887664 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=c8e44abe11554acda26e71b8a11e2e0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.721499 1999 tablet_bootstrap.cc:654] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.721577 32402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26b6d19699047afaee57d6f2c887664. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.722438 1999 tablet_bootstrap.cc:492] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:36.722507 1999 ts_tablet_manager.cc:1367] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:36.722877 1999 raft_consensus.cc:348] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:36.722970 1999 raft_consensus.cc:374] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.723004 1999 raft_consensus.cc:729] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.723116 1999 consensus_queue.cc:260] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:36.723178 1989 consensus_queue.cc:260] T 7b3a90c03de64d6891dda267167d1d7b P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:36.723412 1999 ts_tablet_manager.cc:1398] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.723579 1999 tablet_bootstrap.cc:492] T c90d65f39c3540f48c0d85b28130e76a P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:36.723939 1989 ts_tablet_manager.cc:1398] T 7b3a90c03de64d6891dda267167d1d7b P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:36.724108 1989 tablet_bootstrap.cc:492] T b26b6d19699047afaee57d6f2c887664 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:36.724478 1999 tablet_bootstrap.cc:654] T c90d65f39c3540f48c0d85b28130e76a P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.723733 29870 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 796cef2485c94b76a5f3e1944b73dcf6
I20240501 14:09:36.726300 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 796cef2485c94b76a5f3e1944b73dcf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:36.726380 29870 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 464516e077954ac98413609f05665183
I20240501 14:09:36.726452 543 tablet_replica.cc:330] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:36.726588 543 raft_consensus.cc:2229] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:36.726780 543 raft_consensus.cc:2258] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.727545 1989 tablet_bootstrap.cc:654] T b26b6d19699047afaee57d6f2c887664 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.727519 30256 tablet_service.cc:1514] Processing DeleteTablet for tablet 796cef2485c94b76a5f3e1944b73dcf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:36.728035 1076 consensus_queue.cc:1035] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.728144 543 ts_tablet_manager.cc:1875] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.728830 29870 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet ffcce6d1868242d69287d5831f4233f9
I20240501 14:09:36.729596 29995 tablet_service.cc:1514] Processing DeleteTablet for tablet ffcce6d1868242d69287d5831f4233f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:36.729635 2004 tablet_replica.cc:330] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:36.729741 545 tablet_replica.cc:330] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:36.729774 2004 raft_consensus.cc:2229] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.729835 545 raft_consensus.cc:2229] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:36.729930 2004 raft_consensus.cc:2258] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.729970 545 raft_consensus.cc:2258] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.730129 1999 tablet_bootstrap.cc:492] T c90d65f39c3540f48c0d85b28130e76a P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:36.730163 1997 tablet_bootstrap.cc:492] T 7b3a90c03de64d6891dda267167d1d7b P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:36.730233 1999 ts_tablet_manager.cc:1367] T c90d65f39c3540f48c0d85b28130e76a P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:36.730461 545 ts_tablet_manager.cc:1875] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.730517 543 ts_tablet_manager.cc:1888] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 14:09:36.730571 543 log.cc:1196] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/796cef2485c94b76a5f3e1944b73dcf6
I20240501 14:09:36.730597 1999 raft_consensus.cc:348] T c90d65f39c3540f48c0d85b28130e76a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:36.730690 1999 raft_consensus.cc:374] T c90d65f39c3540f48c0d85b28130e76a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.730720 1999 raft_consensus.cc:729] T c90d65f39c3540f48c0d85b28130e76a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.730804 543 ts_tablet_manager.cc:1909] T 796cef2485c94b76a5f3e1944b73dcf6 P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:36.730814 1999 consensus_queue.cc:260] T c90d65f39c3540f48c0d85b28130e76a P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:36.731061 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 464516e077954ac98413609f05665183 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:36.731143 1999 ts_tablet_manager.cc:1398] T c90d65f39c3540f48c0d85b28130e76a P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.731169 1997 tablet_bootstrap.cc:654] T 7b3a90c03de64d6891dda267167d1d7b P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.732154 1997 tablet_bootstrap.cc:492] T 7b3a90c03de64d6891dda267167d1d7b P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:36.732151 30256 tablet_service.cc:1514] Processing DeleteTablet for tablet ffcce6d1868242d69287d5831f4233f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:36.732239 1997 ts_tablet_manager.cc:1367] T 7b3a90c03de64d6891dda267167d1d7b P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:36.732399 543 tablet_replica.cc:330] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:36.732509 543 raft_consensus.cc:2229] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:36.732671 545 ts_tablet_manager.cc:1888] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:36.732672 543 raft_consensus.cc:2258] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.732731 545 log.cc:1196] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/ffcce6d1868242d69287d5831f4233f9
I20240501 14:09:36.732682 1997 raft_consensus.cc:348] T 7b3a90c03de64d6891dda267167d1d7b P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:36.732770 1997 raft_consensus.cc:374] T 7b3a90c03de64d6891dda267167d1d7b P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.732802 1997 raft_consensus.cc:729] T 7b3a90c03de64d6891dda267167d1d7b P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.732923 1997 consensus_queue.cc:260] T 7b3a90c03de64d6891dda267167d1d7b P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:36.732990 545 ts_tablet_manager.cc:1909] T ffcce6d1868242d69287d5831f4233f9 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:36.733146 543 ts_tablet_manager.cc:1875] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.733191 1997 ts_tablet_manager.cc:1398] T 7b3a90c03de64d6891dda267167d1d7b P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.733325 1997 tablet_bootstrap.cc:492] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:36.734318 1997 tablet_bootstrap.cc:654] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.734308 32664 tablet_service.cc:1467] Processing CreateTablet for tablet b26b6d19699047afaee57d6f2c887664 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=c8e44abe11554acda26e71b8a11e2e0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.734578 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26b6d19699047afaee57d6f2c887664. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.734701 30386 tablet_service.cc:1514] Processing DeleteTablet for tablet ffcce6d1868242d69287d5831f4233f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:36.735494 2004 ts_tablet_manager.cc:1875] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.735742 543 ts_tablet_manager.cc:1888] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:36.735798 543 log.cc:1196] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/464516e077954ac98413609f05665183
I20240501 14:09:36.736193 543 ts_tablet_manager.cc:1909] T 464516e077954ac98413609f05665183 P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:36.736447 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 796cef2485c94b76a5f3e1944b73dcf6 (table rebalance_test_table_16 [id=86d7db413f934f379c4369701c5c13cd]) successfully deleted
I20240501 14:09:36.736757 30388 tablet_service.cc:1514] Processing DeleteTablet for tablet 464516e077954ac98413609f05665183 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:36.737174 32665 tablet_service.cc:1467] Processing CreateTablet for tablet c90d65f39c3540f48c0d85b28130e76a (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=c8e44abe11554acda26e71b8a11e2e0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.737393 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c90d65f39c3540f48c0d85b28130e76a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.737458 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet 464516e077954ac98413609f05665183 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:36.737618 32402 tablet_service.cc:1514] Processing DeleteTablet for tablet 796cef2485c94b76a5f3e1944b73dcf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:36.737773 1943 tablet_replica.cc:330] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:36.737809 1989 tablet_bootstrap.cc:492] T b26b6d19699047afaee57d6f2c887664 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:36.737869 1943 raft_consensus.cc:2229] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.737960 1943 raft_consensus.cc:2258] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.738214 2004 ts_tablet_manager.cc:1888] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 14:09:36.738289 2004 log.cc:1196] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/796cef2485c94b76a5f3e1944b73dcf6
I20240501 14:09:36.738564 2004 ts_tablet_manager.cc:1909] T 796cef2485c94b76a5f3e1944b73dcf6 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting consensus metadata
I20240501 14:09:36.738782 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet ffcce6d1868242d69287d5831f4233f9 (table rebalance_test_table_16 [id=86d7db413f934f379c4369701c5c13cd]) successfully deleted
I20240501 14:09:36.738829 545 tablet_replica.cc:330] T 464516e077954ac98413609f05665183 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:36.739277 545 raft_consensus.cc:2229] T 464516e077954ac98413609f05665183 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.739406 545 raft_consensus.cc:2258] T 464516e077954ac98413609f05665183 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.739373 1989 ts_tablet_manager.cc:1367] T b26b6d19699047afaee57d6f2c887664 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 14:09:36.739866 545 ts_tablet_manager.cc:1875] T 464516e077954ac98413609f05665183 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.740125 1943 ts_tablet_manager.cc:1875] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.740162 1989 raft_consensus.cc:348] T b26b6d19699047afaee57d6f2c887664 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:36.740296 1989 raft_consensus.cc:374] T b26b6d19699047afaee57d6f2c887664 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.740337 1989 raft_consensus.cc:729] T b26b6d19699047afaee57d6f2c887664 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.740388 2004 tablet_replica.cc:330] T ffcce6d1868242d69287d5831f4233f9 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:36.740444 1989 consensus_queue.cc:260] T b26b6d19699047afaee57d6f2c887664 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:36.740494 2004 raft_consensus.cc:2229] T ffcce6d1868242d69287d5831f4233f9 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.740598 2004 raft_consensus.cc:2258] T ffcce6d1868242d69287d5831f4233f9 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.740706 1989 ts_tablet_manager.cc:1398] T b26b6d19699047afaee57d6f2c887664 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.741057 2004 ts_tablet_manager.cc:1875] T ffcce6d1868242d69287d5831f4233f9 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.742147 1943 ts_tablet_manager.cc:1888] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 14:09:36.742208 1943 log.cc:1196] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/796cef2485c94b76a5f3e1944b73dcf6
I20240501 14:09:36.742468 1943 ts_tablet_manager.cc:1909] T 796cef2485c94b76a5f3e1944b73dcf6 P a91029b90bf9415a89a2f70736759ad3: Deleting consensus metadata
I20240501 14:09:36.742693 2004 ts_tablet_manager.cc:1888] T ffcce6d1868242d69287d5831f4233f9 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:36.742786 2004 log.cc:1196] T ffcce6d1868242d69287d5831f4233f9 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/ffcce6d1868242d69287d5831f4233f9
I20240501 14:09:36.742902 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet 796cef2485c94b76a5f3e1944b73dcf6 (table rebalance_test_table_16 [id=86d7db413f934f379c4369701c5c13cd]) successfully deleted
I20240501 14:09:36.742987 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 464516e077954ac98413609f05665183 (table rebalance_test_table_16 [id=86d7db413f934f379c4369701c5c13cd]) successfully deleted
I20240501 14:09:36.743135 2004 ts_tablet_manager.cc:1909] T ffcce6d1868242d69287d5831f4233f9 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting consensus metadata
I20240501 14:09:36.743207 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 796cef2485c94b76a5f3e1944b73dcf6 (table rebalance_test_table_16 [id=86d7db413f934f379c4369701c5c13cd]) successfully deleted
I20240501 14:09:36.743885 2010 tablet_replica.cc:330] T ffcce6d1868242d69287d5831f4233f9 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:36.744055 2010 raft_consensus.cc:2229] T ffcce6d1868242d69287d5831f4233f9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.744091 1997 tablet_bootstrap.cc:492] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:36.744168 1997 ts_tablet_manager.cc:1367] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:09:36.744235 2010 raft_consensus.cc:2258] T ffcce6d1868242d69287d5831f4233f9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.744251 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet ffcce6d1868242d69287d5831f4233f9 (table rebalance_test_table_16 [id=86d7db413f934f379c4369701c5c13cd]) successfully deleted
I20240501 14:09:36.744649 1997 raft_consensus.cc:348] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:36.744724 2010 ts_tablet_manager.cc:1875] T ffcce6d1868242d69287d5831f4233f9 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.744738 1997 raft_consensus.cc:374] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.744767 1997 raft_consensus.cc:729] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.744901 1997 consensus_queue.cc:260] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:36.745160 1997 ts_tablet_manager.cc:1398] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:36.745309 1997 tablet_bootstrap.cc:492] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:36.745507 545 ts_tablet_manager.cc:1888] T 464516e077954ac98413609f05665183 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:36.745620 545 log.cc:1196] T 464516e077954ac98413609f05665183 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/464516e077954ac98413609f05665183
I20240501 14:09:36.745883 545 ts_tablet_manager.cc:1909] T 464516e077954ac98413609f05665183 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:36.746135 1997 tablet_bootstrap.cc:654] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.746536 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 464516e077954ac98413609f05665183 (table rebalance_test_table_16 [id=86d7db413f934f379c4369701c5c13cd]) successfully deleted
I20240501 14:09:36.747879 2010 ts_tablet_manager.cc:1888] T ffcce6d1868242d69287d5831f4233f9 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:36.747996 2010 log.cc:1196] T ffcce6d1868242d69287d5831f4233f9 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/ffcce6d1868242d69287d5831f4233f9
I20240501 14:09:36.748288 2010 ts_tablet_manager.cc:1909] T ffcce6d1868242d69287d5831f4233f9 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:36.748847 2010 tablet_replica.cc:330] T 464516e077954ac98413609f05665183 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:36.748967 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet ffcce6d1868242d69287d5831f4233f9 (table rebalance_test_table_16 [id=86d7db413f934f379c4369701c5c13cd]) successfully deleted
I20240501 14:09:36.749017 2010 raft_consensus.cc:2229] T 464516e077954ac98413609f05665183 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.749197 2010 raft_consensus.cc:2258] T 464516e077954ac98413609f05665183 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.749608 2010 ts_tablet_manager.cc:1875] T 464516e077954ac98413609f05665183 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.751014 2010 ts_tablet_manager.cc:1888] T 464516e077954ac98413609f05665183 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:36.751191 1990 tablet_bootstrap.cc:492] T b26b6d19699047afaee57d6f2c887664 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:36.751601 1997 tablet_bootstrap.cc:492] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:36.751683 1997 ts_tablet_manager.cc:1367] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240501 14:09:36.752053 1990 tablet_bootstrap.cc:654] T b26b6d19699047afaee57d6f2c887664 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.752039 1997 raft_consensus.cc:348] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:36.752121 1997 raft_consensus.cc:374] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.752178 1997 raft_consensus.cc:729] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.752341 1997 consensus_queue.cc:260] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:36.752653 1997 ts_tablet_manager.cc:1398] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:36.752851 2010 log.cc:1196] T 464516e077954ac98413609f05665183 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/464516e077954ac98413609f05665183
I20240501 14:09:36.753131 2010 ts_tablet_manager.cc:1909] T 464516e077954ac98413609f05665183 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:36.753682 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 464516e077954ac98413609f05665183 (table rebalance_test_table_16 [id=86d7db413f934f379c4369701c5c13cd]) successfully deleted
I20240501 14:09:36.756781 1990 tablet_bootstrap.cc:492] T b26b6d19699047afaee57d6f2c887664 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:36.756866 1990 ts_tablet_manager.cc:1367] T b26b6d19699047afaee57d6f2c887664 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240501 14:09:36.757206 1990 raft_consensus.cc:348] T b26b6d19699047afaee57d6f2c887664 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:36.757287 1990 raft_consensus.cc:374] T b26b6d19699047afaee57d6f2c887664 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.757308 1990 raft_consensus.cc:729] T b26b6d19699047afaee57d6f2c887664 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.757431 1990 consensus_queue.cc:260] T b26b6d19699047afaee57d6f2c887664 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:36.757650 1990 ts_tablet_manager.cc:1398] T b26b6d19699047afaee57d6f2c887664 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:36.757862 1990 tablet_bootstrap.cc:492] T c90d65f39c3540f48c0d85b28130e76a P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:36.758949 1990 tablet_bootstrap.cc:654] T c90d65f39c3540f48c0d85b28130e76a P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.760627 1990 tablet_bootstrap.cc:492] T c90d65f39c3540f48c0d85b28130e76a P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:36.760828 1990 ts_tablet_manager.cc:1367] T c90d65f39c3540f48c0d85b28130e76a P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:36.761363 1990 raft_consensus.cc:348] T c90d65f39c3540f48c0d85b28130e76a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:36.761445 1990 raft_consensus.cc:374] T c90d65f39c3540f48c0d85b28130e76a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.761482 1990 raft_consensus.cc:729] T c90d65f39c3540f48c0d85b28130e76a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.761597 1990 consensus_queue.cc:260] T c90d65f39c3540f48c0d85b28130e76a P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:36.761865 1990 ts_tablet_manager.cc:1398] T c90d65f39c3540f48c0d85b28130e76a P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:36.765518 1689 raft_consensus.cc:482] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:36.765602 1689 raft_consensus.cc:504] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:36.766005 1689 leader_election.cc:288] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:36.766245 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3a90c03de64d6891dda267167d1d7b" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:36.766286 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3a90c03de64d6891dda267167d1d7b" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:36.766393 32416 raft_consensus.cc:2454] T 7b3a90c03de64d6891dda267167d1d7b P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:36.766386 32554 raft_consensus.cc:2454] T 7b3a90c03de64d6891dda267167d1d7b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:36.766664 29951 leader_election.cc:302] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:36.766814 1689 raft_consensus.cc:2781] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:36.766875 1689 raft_consensus.cc:482] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:36.766909 1689 raft_consensus.cc:3037] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.767786 1689 raft_consensus.cc:504] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:36.768065 1689 leader_election.cc:288] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:36.768335 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3a90c03de64d6891dda267167d1d7b" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:36.768389 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3a90c03de64d6891dda267167d1d7b" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:36.768443 32554 raft_consensus.cc:3037] T 7b3a90c03de64d6891dda267167d1d7b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.768496 32416 raft_consensus.cc:3037] T 7b3a90c03de64d6891dda267167d1d7b P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.769133 32554 raft_consensus.cc:2454] T 7b3a90c03de64d6891dda267167d1d7b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:36.769153 32416 raft_consensus.cc:2454] T 7b3a90c03de64d6891dda267167d1d7b P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:36.769378 29950 leader_election.cc:302] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:36.769523 1689 raft_consensus.cc:2781] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:36.769593 1689 raft_consensus.cc:686] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:36.769706 1689 consensus_queue.cc:237] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:36.771449 29870 catalog_manager.cc:5484] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:36.781391 562 raft_consensus.cc:482] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:36.781497 562 raft_consensus.cc:504] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:36.781941 562 leader_election.cc:288] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:36.782397 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c90d65f39c3540f48c0d85b28130e76a" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:36.782526 32683 raft_consensus.cc:2454] T c90d65f39c3540f48c0d85b28130e76a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:36.782549 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c90d65f39c3540f48c0d85b28130e76a" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:36.782678 30014 raft_consensus.cc:2454] T c90d65f39c3540f48c0d85b28130e76a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:36.782815 32356 leader_election.cc:302] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:36.782976 562 raft_consensus.cc:2781] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:36.783026 562 raft_consensus.cc:482] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:36.783071 562 raft_consensus.cc:3037] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.783847 562 raft_consensus.cc:504] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:36.784104 562 leader_election.cc:288] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:36.784402 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c90d65f39c3540f48c0d85b28130e76a" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:36.784468 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c90d65f39c3540f48c0d85b28130e76a" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:36.784497 30014 raft_consensus.cc:3037] T c90d65f39c3540f48c0d85b28130e76a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.784575 32683 raft_consensus.cc:3037] T c90d65f39c3540f48c0d85b28130e76a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.785418 30014 raft_consensus.cc:2454] T c90d65f39c3540f48c0d85b28130e76a P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:36.785538 32683 raft_consensus.cc:2454] T c90d65f39c3540f48c0d85b28130e76a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:36.785797 32356 leader_election.cc:302] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:36.785997 562 raft_consensus.cc:2781] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:36.786077 562 raft_consensus.cc:686] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:36.786201 562 consensus_queue.cc:237] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:36.787768 29870 catalog_manager.cc:5484] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:36.874615 562 raft_consensus.cc:482] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:36.874749 562 raft_consensus.cc:504] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:36.875174 562 leader_election.cc:288] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:36.875629 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26b6d19699047afaee57d6f2c887664" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:36.875778 32683 raft_consensus.cc:2454] T b26b6d19699047afaee57d6f2c887664 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:36.875777 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26b6d19699047afaee57d6f2c887664" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:36.875908 32554 raft_consensus.cc:2454] T b26b6d19699047afaee57d6f2c887664 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:36.876091 32356 leader_election.cc:302] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:36.876256 562 raft_consensus.cc:2781] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:36.876314 562 raft_consensus.cc:482] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:36.876358 562 raft_consensus.cc:3037] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.877233 562 raft_consensus.cc:504] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:36.877544 562 leader_election.cc:288] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:36.877806 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26b6d19699047afaee57d6f2c887664" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:36.877833 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26b6d19699047afaee57d6f2c887664" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:36.877929 32553 raft_consensus.cc:3037] T b26b6d19699047afaee57d6f2c887664 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.877930 32683 raft_consensus.cc:3037] T b26b6d19699047afaee57d6f2c887664 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.878680 32553 raft_consensus.cc:2454] T b26b6d19699047afaee57d6f2c887664 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:36.878722 32683 raft_consensus.cc:2454] T b26b6d19699047afaee57d6f2c887664 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:36.878979 32356 leader_election.cc:302] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:36.879134 562 raft_consensus.cc:2781] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:36.879228 562 raft_consensus.cc:686] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:36.879372 562 consensus_queue.cc:237] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:36.880838 29872 catalog_manager.cc:5484] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:36.988384 30406 raft_consensus.cc:1261] T 3b6a526c2ed84e1780ec392ef39d09b0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:36.988883 2003 consensus_queue.cc:1035] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.046090 1548 raft_consensus.cc:482] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:37.046211 1548 raft_consensus.cc:504] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:37.046633 1548 leader_election.cc:288] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:37.046940 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52acda15aaec4b6fbab788e8bbe0ff30" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:37.046965 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52acda15aaec4b6fbab788e8bbe0ff30" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:37.047062 30146 raft_consensus.cc:2454] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:37.047097 32683 raft_consensus.cc:2454] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:37.047325 32490 leader_election.cc:302] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:37.047502 1548 raft_consensus.cc:2781] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:37.047560 1548 raft_consensus.cc:482] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:37.047597 1548 raft_consensus.cc:3037] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.048565 1548 raft_consensus.cc:504] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:37.048913 1548 leader_election.cc:288] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:37.049237 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52acda15aaec4b6fbab788e8bbe0ff30" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:37.049343 32683 raft_consensus.cc:3037] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.049338 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52acda15aaec4b6fbab788e8bbe0ff30" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:37.049450 30146 raft_consensus.cc:3037] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.050288 32683 raft_consensus.cc:2454] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:37.050432 30146 raft_consensus.cc:2454] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:37.050587 32487 leader_election.cc:302] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:37.050776 1548 raft_consensus.cc:2781] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:37.050912 1548 raft_consensus.cc:686] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:09:37.051061 1548 consensus_queue.cc:237] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:37.052855 29872 catalog_manager.cc:5484] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:37.075363 30256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:37.076975 329 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:37.081141 32534 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:37.083351 29993 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:37.089133 30388 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:37.098973 32665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:37.099711 32402 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:37.109302 30125 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:37.164747 1996 consensus_queue.cc:1035] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.168284 1996 consensus_queue.cc:1035] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.170672 30146 consensus_queue.cc:237] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:37.171967 30264 raft_consensus.cc:1261] T fc28b2028a9246b99e8887f03180d0c3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.172041 32683 raft_consensus.cc:1261] T fc28b2028a9246b99e8887f03180d0c3 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.172418 2003 consensus_queue.cc:1035] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.172731 2003 consensus_queue.cc:1035] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.174018 2003 raft_consensus.cc:2932] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:37.174414 30264 raft_consensus.cc:2932] T fc28b2028a9246b99e8887f03180d0c3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:37.176287 32683 raft_consensus.cc:2932] T fc28b2028a9246b99e8887f03180d0c3 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:37.177749 29871 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:37.178825 29868 catalog_manager.cc:5484] T fc28b2028a9246b99e8887f03180d0c3 P db935501e9104de5b15c8780681d6c38 reported cstate change: config changed from index 4 to 5, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:37.182585 30146 raft_consensus.cc:1261] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:37.182633 32683 raft_consensus.cc:1261] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:37.183241 30388 tablet_service.cc:1514] Processing DeleteTablet for tablet fc28b2028a9246b99e8887f03180d0c3 with delete_type TABLET_DATA_TOMBSTONED (TS d1c217469ae4478ebd49e4ed9f7352d7 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:37.183516 2058 consensus_queue.cc:1035] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.183743 1548 consensus_queue.cc:1035] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.184360 2010 tablet_replica.cc:330] T fc28b2028a9246b99e8887f03180d0c3 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:37.184497 2010 raft_consensus.cc:2229] T fc28b2028a9246b99e8887f03180d0c3 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.184638 2010 raft_consensus.cc:2258] T fc28b2028a9246b99e8887f03180d0c3 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.185164 2010 ts_tablet_manager.cc:1875] T fc28b2028a9246b99e8887f03180d0c3 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:37.186398 29867 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet 52acda15aaec4b6fbab788e8bbe0ff30 (table rebalancer_dynamic_table [id=593b9102cd8c4770a852a8a5ac1db30d]). Expected version 1 got 0
I20240501 14:09:37.187156 2100 tablet.cc:1722] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:37.187438 2010 ts_tablet_manager.cc:1888] T fc28b2028a9246b99e8887f03180d0c3 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:37.187685 2100 tablet.cc:1620] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:37.188057 2010 log.cc:1196] T fc28b2028a9246b99e8887f03180d0c3 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/fc28b2028a9246b99e8887f03180d0c3
I20240501 14:09:37.188102 2101 tablet.cc:1722] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:37.188654 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet fc28b2028a9246b99e8887f03180d0c3 (table rebalance_test_table_7 [id=8f3109027b0c482d89eb86da7f7ed641]) successfully deleted
I20240501 14:09:37.188853 2101 tablet.cc:1620] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3: MemRowSet was empty: no flush needed.
I20240501 14:09:37.193048 348 raft_consensus.cc:1261] T 3b6a526c2ed84e1780ec392ef39d09b0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:37.193933 2003 consensus_queue.cc:1035] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.195282 29867 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=593b9102cd8c4770a852a8a5ac1db30d] alter complete (version 1)
I20240501 14:09:37.198899 2104 tablet.cc:1722] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:37.198994 29872 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet 52acda15aaec4b6fbab788e8bbe0ff30 (table rebalancer_dynamic_table [id=593b9102cd8c4770a852a8a5ac1db30d]). Expected version 1 got 0
I20240501 14:09:37.200067 30264 raft_consensus.cc:1261] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:37.200465 2003 consensus_queue.cc:1035] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.202891 2104 tablet.cc:1620] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:37.235296 29868 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 14:09:37.235453 29868 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 14:09:37.237818 29868 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 727bad29bdb44df3b032de5d51c2b7fd
I20240501 14:09:37.238114 29868 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 217878532a4642179f06645f3e073a23
I20240501 14:09:37.238415 29868 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 4 replicas of tablet 205a0fda164d46a29a031070f1d0c369
I20240501 14:09:37.238399 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet 727bad29bdb44df3b032de5d51c2b7fd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:37.238577 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 727bad29bdb44df3b032de5d51c2b7fd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:37.238616 30256 tablet_service.cc:1514] Processing DeleteTablet for tablet 217878532a4642179f06645f3e073a23 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:37.238782 2004 tablet_replica.cc:330] T 217878532a4642179f06645f3e073a23 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:37.238855 545 tablet_replica.cc:330] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:37.238899 2125 tablet_replica.cc:330] T 727bad29bdb44df3b032de5d51c2b7fd P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:37.238915 2004 raft_consensus.cc:2229] T 217878532a4642179f06645f3e073a23 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.238976 545 raft_consensus.cc:2229] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:37.239054 2004 raft_consensus.cc:2258] T 217878532a4642179f06645f3e073a23 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.239049 2125 raft_consensus.cc:2229] T 727bad29bdb44df3b032de5d51c2b7fd P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.239182 2125 raft_consensus.cc:2258] T 727bad29bdb44df3b032de5d51c2b7fd P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.239183 545 raft_consensus.cc:2258] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.239557 2004 ts_tablet_manager.cc:1875] T 217878532a4642179f06645f3e073a23 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.239625 545 ts_tablet_manager.cc:1875] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.239787 2125 ts_tablet_manager.cc:1875] T 727bad29bdb44df3b032de5d51c2b7fd P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.239964 30386 tablet_service.cc:1514] Processing DeleteTablet for tablet 217878532a4642179f06645f3e073a23 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:37.240140 2010 tablet_replica.cc:330] T 217878532a4642179f06645f3e073a23 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:37.240306 2010 raft_consensus.cc:2229] T 217878532a4642179f06645f3e073a23 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.240456 2010 raft_consensus.cc:2258] T 217878532a4642179f06645f3e073a23 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.240988 2010 ts_tablet_manager.cc:1875] T 217878532a4642179f06645f3e073a23 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.241175 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet 217878532a4642179f06645f3e073a23 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:37.241928 2004 ts_tablet_manager.cc:1888] T 217878532a4642179f06645f3e073a23 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:37.242000 2004 log.cc:1196] T 217878532a4642179f06645f3e073a23 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/217878532a4642179f06645f3e073a23
I20240501 14:09:37.242269 2004 ts_tablet_manager.cc:1909] T 217878532a4642179f06645f3e073a23 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting consensus metadata
I20240501 14:09:37.242460 30388 tablet_service.cc:1514] Processing DeleteTablet for tablet 727bad29bdb44df3b032de5d51c2b7fd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:37.242271 2125 ts_tablet_manager.cc:1888] T 727bad29bdb44df3b032de5d51c2b7fd P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:37.242588 2125 log.cc:1196] T 727bad29bdb44df3b032de5d51c2b7fd P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/727bad29bdb44df3b032de5d51c2b7fd
I20240501 14:09:37.242581 30387 tablet_service.cc:1514] Processing DeleteTablet for tablet 205a0fda164d46a29a031070f1d0c369 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:37.242851 2125 ts_tablet_manager.cc:1909] T 727bad29bdb44df3b032de5d51c2b7fd P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:37.243311 2010 ts_tablet_manager.cc:1888] T 217878532a4642179f06645f3e073a23 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:37.243336 545 ts_tablet_manager.cc:1888] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:37.243382 2010 log.cc:1196] T 217878532a4642179f06645f3e073a23 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/217878532a4642179f06645f3e073a23
I20240501 14:09:37.243400 545 log.cc:1196] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/727bad29bdb44df3b032de5d51c2b7fd
I20240501 14:09:37.243659 545 ts_tablet_manager.cc:1909] T 727bad29bdb44df3b032de5d51c2b7fd P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:37.243918 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 727bad29bdb44df3b032de5d51c2b7fd (table rebalance_test_table_18 [id=e756cf3e07134acfb79003690692e338]) successfully deleted
I20240501 14:09:37.244347 32534 tablet_service.cc:1514] Processing DeleteTablet for tablet 205a0fda164d46a29a031070f1d0c369 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:37.244485 545 tablet_replica.cc:330] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:37.244632 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 727bad29bdb44df3b032de5d51c2b7fd (table rebalance_test_table_18 [id=e756cf3e07134acfb79003690692e338]) successfully deleted
I20240501 14:09:37.244731 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 205a0fda164d46a29a031070f1d0c369 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:37.244920 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet 205a0fda164d46a29a031070f1d0c369 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:37.245119 545 raft_consensus.cc:2229] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:37.245165 2010 ts_tablet_manager.cc:1909] T 217878532a4642179f06645f3e073a23 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:37.245375 545 raft_consensus.cc:2258] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.245728 2010 tablet_replica.cc:330] T 727bad29bdb44df3b032de5d51c2b7fd P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:37.245864 2010 raft_consensus.cc:2229] T 727bad29bdb44df3b032de5d51c2b7fd P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.245882 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 217878532a4642179f06645f3e073a23 (table rebalance_test_table_18 [id=e756cf3e07134acfb79003690692e338]) successfully deleted
I20240501 14:09:37.245991 2010 raft_consensus.cc:2258] T 727bad29bdb44df3b032de5d51c2b7fd P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.245994 545 ts_tablet_manager.cc:1875] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.246026 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet 217878532a4642179f06645f3e073a23 (table rebalance_test_table_18 [id=e756cf3e07134acfb79003690692e338]) successfully deleted
I20240501 14:09:37.246294 2126 tablet_replica.cc:330] T 205a0fda164d46a29a031070f1d0c369 P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:37.246364 2010 ts_tablet_manager.cc:1875] T 727bad29bdb44df3b032de5d51c2b7fd P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.246417 2126 raft_consensus.cc:2229] T 205a0fda164d46a29a031070f1d0c369 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.246517 2126 raft_consensus.cc:2258] T 205a0fda164d46a29a031070f1d0c369 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.247011 2126 ts_tablet_manager.cc:1875] T 205a0fda164d46a29a031070f1d0c369 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.247747 2010 ts_tablet_manager.cc:1888] T 727bad29bdb44df3b032de5d51c2b7fd P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:37.247826 2010 log.cc:1196] T 727bad29bdb44df3b032de5d51c2b7fd P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/727bad29bdb44df3b032de5d51c2b7fd
I20240501 14:09:37.247817 545 ts_tablet_manager.cc:1888] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:37.247875 545 log.cc:1196] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/217878532a4642179f06645f3e073a23
I20240501 14:09:37.248080 2010 ts_tablet_manager.cc:1909] T 727bad29bdb44df3b032de5d51c2b7fd P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:37.248200 545 ts_tablet_manager.cc:1909] T 217878532a4642179f06645f3e073a23 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:37.248600 2010 tablet_replica.cc:330] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:37.248752 2010 raft_consensus.cc:2229] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:37.248739 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 727bad29bdb44df3b032de5d51c2b7fd (table rebalance_test_table_18 [id=e756cf3e07134acfb79003690692e338]) successfully deleted
I20240501 14:09:37.248857 545 tablet_replica.cc:330] T 205a0fda164d46a29a031070f1d0c369 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:37.248926 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 217878532a4642179f06645f3e073a23 (table rebalance_test_table_18 [id=e756cf3e07134acfb79003690692e338]) successfully deleted
I20240501 14:09:37.249007 2010 raft_consensus.cc:2258] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.249053 545 raft_consensus.cc:2229] T 205a0fda164d46a29a031070f1d0c369 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.249294 545 raft_consensus.cc:2258] T 205a0fda164d46a29a031070f1d0c369 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.249354 2126 ts_tablet_manager.cc:1888] T 205a0fda164d46a29a031070f1d0c369 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:37.249455 2126 log.cc:1196] T 205a0fda164d46a29a031070f1d0c369 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/205a0fda164d46a29a031070f1d0c369
I20240501 14:09:37.249505 2010 ts_tablet_manager.cc:1875] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.249831 2126 ts_tablet_manager.cc:1909] T 205a0fda164d46a29a031070f1d0c369 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:37.249869 545 ts_tablet_manager.cc:1875] T 205a0fda164d46a29a031070f1d0c369 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.250532 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet 205a0fda164d46a29a031070f1d0c369 (table rebalance_test_table_18 [id=e756cf3e07134acfb79003690692e338]) successfully deleted
I20240501 14:09:37.250721 2125 tablet_replica.cc:330] T 205a0fda164d46a29a031070f1d0c369 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:37.251441 2010 ts_tablet_manager.cc:1888] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:37.251518 2010 log.cc:1196] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/205a0fda164d46a29a031070f1d0c369
I20240501 14:09:37.251772 2010 ts_tablet_manager.cc:1909] T 205a0fda164d46a29a031070f1d0c369 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:37.252027 2125 raft_consensus.cc:2229] T 205a0fda164d46a29a031070f1d0c369 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.252266 2125 raft_consensus.cc:2258] T 205a0fda164d46a29a031070f1d0c369 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.252414 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 205a0fda164d46a29a031070f1d0c369 (table rebalance_test_table_18 [id=e756cf3e07134acfb79003690692e338]) successfully deleted
I20240501 14:09:37.252737 2125 ts_tablet_manager.cc:1875] T 205a0fda164d46a29a031070f1d0c369 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.254788 2125 ts_tablet_manager.cc:1888] T 205a0fda164d46a29a031070f1d0c369 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:37.254885 545 ts_tablet_manager.cc:1888] T 205a0fda164d46a29a031070f1d0c369 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:37.255000 545 log.cc:1196] T 205a0fda164d46a29a031070f1d0c369 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/205a0fda164d46a29a031070f1d0c369
I20240501 14:09:37.255312 545 ts_tablet_manager.cc:1909] T 205a0fda164d46a29a031070f1d0c369 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:37.255844 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 205a0fda164d46a29a031070f1d0c369 (table rebalance_test_table_18 [id=e756cf3e07134acfb79003690692e338]) successfully deleted
I20240501 14:09:37.255925 2125 log.cc:1196] T 205a0fda164d46a29a031070f1d0c369 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/205a0fda164d46a29a031070f1d0c369
I20240501 14:09:37.256223 2125 ts_tablet_manager.cc:1909] T 205a0fda164d46a29a031070f1d0c369 P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:37.256919 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 205a0fda164d46a29a031070f1d0c369 (table rebalance_test_table_18 [id=e756cf3e07134acfb79003690692e338]) successfully deleted
I20240501 14:09:37.259352 562 consensus_queue.cc:1035] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.263046 562 consensus_queue.cc:1035] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.292282 30256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:37.293298 29993 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:37.295624 32534 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:37.297092 30387 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:37.304436 30125 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:37.304891 29868 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:37.313813 2100 tablet.cc:1722] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:37.314100 2100 tablet.cc:1620] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:37.315524 329 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:37.316931 2101 tablet.cc:1722] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:37.317225 2101 tablet.cc:1620] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3: MemRowSet was empty: no flush needed.
I20240501 14:09:37.318239 32402 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:37.319690 2104 tablet.cc:1722] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:37.319939 2104 tablet.cc:1620] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:37.327580 32665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:37.327520 29868 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=593b9102cd8c4770a852a8a5ac1db30d] alter complete (version 2)
I20240501 14:09:37.374724 30146 consensus_queue.cc:237] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:37.375907 348 raft_consensus.cc:1261] T 8d87b2a4f6764e20afbfb43681f58467 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.375972 30013 raft_consensus.cc:1261] T 8d87b2a4f6764e20afbfb43681f58467 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.376334 2003 consensus_queue.cc:1035] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.376521 1076 consensus_queue.cc:1035] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.378371 2002 raft_consensus.cc:2932] T 8d87b2a4f6764e20afbfb43681f58467 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:37.378687 348 raft_consensus.cc:2932] T 8d87b2a4f6764e20afbfb43681f58467 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:37.378732 30013 raft_consensus.cc:2932] T 8d87b2a4f6764e20afbfb43681f58467 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:37.382437 29868 catalog_manager.cc:5484] T 8d87b2a4f6764e20afbfb43681f58467 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:37.385365 30256 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d87b2a4f6764e20afbfb43681f58467 with delete_type TABLET_DATA_TOMBSTONED (TS 1d0bb69611af4876a2b3b33c4766e13a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:37.385684 2004 tablet_replica.cc:330] T 8d87b2a4f6764e20afbfb43681f58467 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:37.385818 2004 raft_consensus.cc:2229] T 8d87b2a4f6764e20afbfb43681f58467 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.385921 2004 raft_consensus.cc:2258] T 8d87b2a4f6764e20afbfb43681f58467 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.386381 2004 ts_tablet_manager.cc:1875] T 8d87b2a4f6764e20afbfb43681f58467 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:37.388166 2004 ts_tablet_manager.cc:1888] T 8d87b2a4f6764e20afbfb43681f58467 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:37.388254 2004 log.cc:1196] T 8d87b2a4f6764e20afbfb43681f58467 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/8d87b2a4f6764e20afbfb43681f58467
I20240501 14:09:37.388986 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet 8d87b2a4f6764e20afbfb43681f58467 (table rebalance_test_table_7 [id=8f3109027b0c482d89eb86da7f7ed641]) successfully deleted
I20240501 14:09:37.389529 29872 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:37.394809 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 1e03e76474584273ae050595a2286559 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=dcad19a5572a46a992d9c7fd61cfcf2b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.395123 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e03e76474584273ae050595a2286559. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.395162 29993 tablet_service.cc:1467] Processing CreateTablet for tablet b66c71cbccfa4c96bdf2b438938bf56b (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=dcad19a5572a46a992d9c7fd61cfcf2b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.395411 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b66c71cbccfa4c96bdf2b438938bf56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.395467 329 tablet_service.cc:1467] Processing CreateTablet for tablet b66c71cbccfa4c96bdf2b438938bf56b (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=dcad19a5572a46a992d9c7fd61cfcf2b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.395720 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b66c71cbccfa4c96bdf2b438938bf56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.397719 328 tablet_service.cc:1467] Processing CreateTablet for tablet 35dd2c9d436448c482ef0b3fac3a3b64 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=dcad19a5572a46a992d9c7fd61cfcf2b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.397917 328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35dd2c9d436448c482ef0b3fac3a3b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.397866 32533 tablet_service.cc:1467] Processing CreateTablet for tablet b66c71cbccfa4c96bdf2b438938bf56b (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=dcad19a5572a46a992d9c7fd61cfcf2b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.397961 2190 tablet_bootstrap.cc:492] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:37.397919 32402 tablet_service.cc:1467] Processing CreateTablet for tablet 1e03e76474584273ae050595a2286559 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=dcad19a5572a46a992d9c7fd61cfcf2b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.398156 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b66c71cbccfa4c96bdf2b438938bf56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.398173 32402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e03e76474584273ae050595a2286559. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.398967 2190 tablet_bootstrap.cc:654] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.400002 32532 tablet_service.cc:1467] Processing CreateTablet for tablet 35dd2c9d436448c482ef0b3fac3a3b64 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=dcad19a5572a46a992d9c7fd61cfcf2b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.400287 32532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35dd2c9d436448c482ef0b3fac3a3b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.400338 2192 tablet_bootstrap.cc:492] T b66c71cbccfa4c96bdf2b438938bf56b P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:37.400622 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 1e03e76474584273ae050595a2286559 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=dcad19a5572a46a992d9c7fd61cfcf2b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.400844 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e03e76474584273ae050595a2286559. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.401152 2190 tablet_bootstrap.cc:492] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:37.401211 2190 ts_tablet_manager.cc:1367] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:37.401690 2190 raft_consensus.cc:348] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:37.401765 2190 raft_consensus.cc:374] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.401785 2190 raft_consensus.cc:729] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.401883 2190 consensus_queue.cc:260] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:37.402132 2190 ts_tablet_manager.cc:1398] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.402916 2192 tablet_bootstrap.cc:654] T b66c71cbccfa4c96bdf2b438938bf56b P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.403388 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 35dd2c9d436448c482ef0b3fac3a3b64 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=dcad19a5572a46a992d9c7fd61cfcf2b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.403592 2193 tablet_bootstrap.cc:492] T 1e03e76474584273ae050595a2286559 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:37.403734 2188 tablet_bootstrap.cc:492] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:37.404644 2188 tablet_bootstrap.cc:654] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.405277 2189 tablet_bootstrap.cc:492] T b66c71cbccfa4c96bdf2b438938bf56b P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:37.405531 562 consensus_queue.cc:1035] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.405860 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35dd2c9d436448c482ef0b3fac3a3b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.406147 2189 tablet_bootstrap.cc:654] T b66c71cbccfa4c96bdf2b438938bf56b P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.407117 2188 tablet_bootstrap.cc:492] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:37.407186 2188 ts_tablet_manager.cc:1367] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:37.405625 2190 tablet_bootstrap.cc:492] T 35dd2c9d436448c482ef0b3fac3a3b64 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:37.407617 2188 raft_consensus.cc:348] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:37.407709 2188 raft_consensus.cc:374] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.407747 2188 raft_consensus.cc:729] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.407882 2188 consensus_queue.cc:260] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:37.408339 2188 ts_tablet_manager.cc:1398] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:37.408409 2190 tablet_bootstrap.cc:654] T 35dd2c9d436448c482ef0b3fac3a3b64 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.412850 1253 consensus_queue.cc:1035] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.414547 2193 tablet_bootstrap.cc:654] T 1e03e76474584273ae050595a2286559 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.416817 2189 tablet_bootstrap.cc:492] T b66c71cbccfa4c96bdf2b438938bf56b P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:37.416906 2189 ts_tablet_manager.cc:1367] T b66c71cbccfa4c96bdf2b438938bf56b P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:37.416972 2190 tablet_bootstrap.cc:492] T 35dd2c9d436448c482ef0b3fac3a3b64 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:37.417043 2190 ts_tablet_manager.cc:1367] T 35dd2c9d436448c482ef0b3fac3a3b64 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240501 14:09:37.417376 2189 raft_consensus.cc:348] T b66c71cbccfa4c96bdf2b438938bf56b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:37.417475 2189 raft_consensus.cc:374] T b66c71cbccfa4c96bdf2b438938bf56b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.417454 2190 raft_consensus.cc:348] T 35dd2c9d436448c482ef0b3fac3a3b64 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:37.417512 2189 raft_consensus.cc:729] T b66c71cbccfa4c96bdf2b438938bf56b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.417548 2190 raft_consensus.cc:374] T 35dd2c9d436448c482ef0b3fac3a3b64 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.417578 2190 raft_consensus.cc:729] T 35dd2c9d436448c482ef0b3fac3a3b64 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.417625 2189 consensus_queue.cc:260] T b66c71cbccfa4c96bdf2b438938bf56b P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:37.417688 2190 consensus_queue.cc:260] T 35dd2c9d436448c482ef0b3fac3a3b64 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:37.417904 2189 ts_tablet_manager.cc:1398] T b66c71cbccfa4c96bdf2b438938bf56b P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.417932 2190 ts_tablet_manager.cc:1398] T 35dd2c9d436448c482ef0b3fac3a3b64 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:37.418962 2192 tablet_bootstrap.cc:492] T b66c71cbccfa4c96bdf2b438938bf56b P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:37.419047 2192 ts_tablet_manager.cc:1367] T b66c71cbccfa4c96bdf2b438938bf56b P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 14:09:37.419440 2192 raft_consensus.cc:348] T b66c71cbccfa4c96bdf2b438938bf56b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:37.419570 2192 raft_consensus.cc:374] T b66c71cbccfa4c96bdf2b438938bf56b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.419610 2192 raft_consensus.cc:729] T b66c71cbccfa4c96bdf2b438938bf56b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.419704 2192 consensus_queue.cc:260] T b66c71cbccfa4c96bdf2b438938bf56b P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:37.419927 2192 ts_tablet_manager.cc:1398] T b66c71cbccfa4c96bdf2b438938bf56b P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.420115 2192 tablet_bootstrap.cc:492] T 35dd2c9d436448c482ef0b3fac3a3b64 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:37.421070 2192 tablet_bootstrap.cc:654] T 35dd2c9d436448c482ef0b3fac3a3b64 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.422556 640 raft_consensus.cc:482] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:37.422624 640 raft_consensus.cc:504] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:37.422951 640 leader_election.cc:288] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:37.423225 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66c71cbccfa4c96bdf2b438938bf56b" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:37.423328 32553 raft_consensus.cc:2454] T b66c71cbccfa4c96bdf2b438938bf56b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:37.423599 2193 tablet_bootstrap.cc:492] T 1e03e76474584273ae050595a2286559 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:37.423581 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66c71cbccfa4c96bdf2b438938bf56b" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:37.423663 2193 ts_tablet_manager.cc:1367] T 1e03e76474584273ae050595a2286559 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 14:09:37.423672 30013 raft_consensus.cc:2454] T b66c71cbccfa4c96bdf2b438938bf56b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:37.423898 32752 leader_election.cc:302] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:37.424078 2193 raft_consensus.cc:348] T 1e03e76474584273ae050595a2286559 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:37.424176 2193 raft_consensus.cc:374] T 1e03e76474584273ae050595a2286559 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.424206 2193 raft_consensus.cc:729] T 1e03e76474584273ae050595a2286559 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.424214 640 raft_consensus.cc:2781] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:37.424284 640 raft_consensus.cc:482] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:37.424319 640 raft_consensus.cc:3037] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.424331 2193 consensus_queue.cc:260] T 1e03e76474584273ae050595a2286559 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:37.424563 2193 ts_tablet_manager.cc:1398] T 1e03e76474584273ae050595a2286559 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.424700 2193 tablet_bootstrap.cc:492] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:37.425138 640 raft_consensus.cc:504] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:37.425436 640 leader_election.cc:288] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:37.425611 2193 tablet_bootstrap.cc:654] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.426393 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66c71cbccfa4c96bdf2b438938bf56b" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:37.426476 30013 raft_consensus.cc:3037] T b66c71cbccfa4c96bdf2b438938bf56b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.426620 2193 tablet_bootstrap.cc:492] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:37.426687 2193 ts_tablet_manager.cc:1367] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:37.427103 2193 raft_consensus.cc:348] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:37.427276 2193 raft_consensus.cc:374] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.427312 30013 raft_consensus.cc:2454] T b66c71cbccfa4c96bdf2b438938bf56b P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:37.427353 2193 raft_consensus.cc:729] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.427583 2193 consensus_queue.cc:260] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:37.427819 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66c71cbccfa4c96bdf2b438938bf56b" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:37.427932 32553 raft_consensus.cc:3037] T b66c71cbccfa4c96bdf2b438938bf56b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.428035 2193 ts_tablet_manager.cc:1398] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.428776 32553 raft_consensus.cc:2454] T b66c71cbccfa4c96bdf2b438938bf56b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:37.429203 32752 leader_election.cc:302] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:37.429497 640 raft_consensus.cc:2781] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:37.429654 640 raft_consensus.cc:686] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:37.429765 640 consensus_queue.cc:237] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:37.431053 2192 tablet_bootstrap.cc:492] T 35dd2c9d436448c482ef0b3fac3a3b64 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:37.431136 2192 ts_tablet_manager.cc:1367] T 35dd2c9d436448c482ef0b3fac3a3b64 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:09:37.431108 29866 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:37.431555 2192 raft_consensus.cc:348] T 35dd2c9d436448c482ef0b3fac3a3b64 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:37.431663 2192 raft_consensus.cc:374] T 35dd2c9d436448c482ef0b3fac3a3b64 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.431692 2192 raft_consensus.cc:729] T 35dd2c9d436448c482ef0b3fac3a3b64 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.431638 29868 catalog_manager.cc:5484] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:37.434098 2192 consensus_queue.cc:260] T 35dd2c9d436448c482ef0b3fac3a3b64 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:37.434836 2192 ts_tablet_manager.cc:1398] T 35dd2c9d436448c482ef0b3fac3a3b64 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:37.435050 2192 tablet_bootstrap.cc:492] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:37.438292 2192 tablet_bootstrap.cc:654] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.438779 2101 tablet.cc:1722] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:37.439764 2104 tablet.cc:1722] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:37.439153 2100 tablet.cc:1722] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:37.445029 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=593b9102cd8c4770a852a8a5ac1db30d] alter complete (version 3)
I20240501 14:09:37.445115 2192 tablet_bootstrap.cc:492] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:37.445271 2192 ts_tablet_manager.cc:1367] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:37.445746 2192 raft_consensus.cc:348] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:37.445827 2192 raft_consensus.cc:374] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.445859 2192 raft_consensus.cc:729] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.445971 2192 consensus_queue.cc:260] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:37.446238 2192 ts_tablet_manager.cc:1398] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.515405 1253 raft_consensus.cc:482] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:37.515547 1253 raft_consensus.cc:504] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:37.515936 1253 leader_election.cc:288] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:37.516338 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35dd2c9d436448c482ef0b3fac3a3b64" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:37.516503 348 raft_consensus.cc:2454] T 35dd2c9d436448c482ef0b3fac3a3b64 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:37.516491 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35dd2c9d436448c482ef0b3fac3a3b64" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:37.516615 32553 raft_consensus.cc:2454] T 35dd2c9d436448c482ef0b3fac3a3b64 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:37.516860 32358 leader_election.cc:302] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:37.517060 1253 raft_consensus.cc:2781] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:37.517112 1253 raft_consensus.cc:482] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:37.517145 1253 raft_consensus.cc:3037] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.518193 1253 raft_consensus.cc:504] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:37.518536 1253 leader_election.cc:288] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:37.518805 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35dd2c9d436448c482ef0b3fac3a3b64" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:37.518918 348 raft_consensus.cc:3037] T 35dd2c9d436448c482ef0b3fac3a3b64 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.519062 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35dd2c9d436448c482ef0b3fac3a3b64" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:37.519162 32553 raft_consensus.cc:3037] T 35dd2c9d436448c482ef0b3fac3a3b64 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.519862 32553 raft_consensus.cc:2454] T 35dd2c9d436448c482ef0b3fac3a3b64 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:37.519971 348 raft_consensus.cc:2454] T 35dd2c9d436448c482ef0b3fac3a3b64 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:37.520140 32359 leader_election.cc:302] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:37.520315 1253 raft_consensus.cc:2781] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:37.520397 1253 raft_consensus.cc:686] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:37.520550 1253 consensus_queue.cc:237] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:37.522035 29867 catalog_manager.cc:5484] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:37.548405 29867 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:37.548561 29867 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:37.550562 29867 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 52acda15aaec4b6fbab788e8bbe0ff30
I20240501 14:09:37.551054 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 52acda15aaec4b6fbab788e8bbe0ff30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:37.551152 32534 tablet_service.cc:1514] Processing DeleteTablet for tablet 52acda15aaec4b6fbab788e8bbe0ff30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:37.551244 2125 tablet_replica.cc:330] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:37.551347 32665 tablet_service.cc:1514] Processing DeleteTablet for tablet 52acda15aaec4b6fbab788e8bbe0ff30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:37.551374 2125 raft_consensus.cc:2229] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.551473 2125 raft_consensus.cc:2258] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.551671 2126 tablet_replica.cc:330] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:37.551836 2126 raft_consensus.cc:2229] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:37.551981 2236 tablet_replica.cc:330] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:37.552062 2126 raft_consensus.cc:2258] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.552129 2125 ts_tablet_manager.cc:1875] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.552109 2236 raft_consensus.cc:2229] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.552333 2236 raft_consensus.cc:2258] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.552942 2236 ts_tablet_manager.cc:1875] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.552997 2126 ts_tablet_manager.cc:1875] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.554644 2125 ts_tablet_manager.cc:1888] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:37.554708 2125 log.cc:1196] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/52acda15aaec4b6fbab788e8bbe0ff30
I20240501 14:09:37.554863 2126 ts_tablet_manager.cc:1888] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:37.554931 2126 log.cc:1196] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/52acda15aaec4b6fbab788e8bbe0ff30
I20240501 14:09:37.554991 2125 ts_tablet_manager.cc:1909] T 52acda15aaec4b6fbab788e8bbe0ff30 P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:37.555182 2126 ts_tablet_manager.cc:1909] T 52acda15aaec4b6fbab788e8bbe0ff30 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:37.555361 2236 ts_tablet_manager.cc:1888] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:37.555423 2236 log.cc:1196] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/52acda15aaec4b6fbab788e8bbe0ff30
I20240501 14:09:37.555680 2236 ts_tablet_manager.cc:1909] T 52acda15aaec4b6fbab788e8bbe0ff30 P db935501e9104de5b15c8780681d6c38: Deleting consensus metadata
I20240501 14:09:37.555765 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet 52acda15aaec4b6fbab788e8bbe0ff30 (table rebalancer_dynamic_table_new_name [id=593b9102cd8c4770a852a8a5ac1db30d]) successfully deleted
I20240501 14:09:37.555938 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 52acda15aaec4b6fbab788e8bbe0ff30 (table rebalancer_dynamic_table_new_name [id=593b9102cd8c4770a852a8a5ac1db30d]) successfully deleted
I20240501 14:09:37.556325 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet 52acda15aaec4b6fbab788e8bbe0ff30 (table rebalancer_dynamic_table_new_name [id=593b9102cd8c4770a852a8a5ac1db30d]) successfully deleted
I20240501 14:09:37.615309 1548 raft_consensus.cc:482] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:37.615444 1548 raft_consensus.cc:504] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:37.615913 1548 leader_election.cc:288] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:37.616286 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e03e76474584273ae050595a2286559" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:37.616443 32683 raft_consensus.cc:2454] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:37.616791 32487 leader_election.cc:302] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:37.616966 1548 raft_consensus.cc:2781] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:37.617040 1548 raft_consensus.cc:482] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:37.617081 1548 raft_consensus.cc:3037] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.618139 1548 raft_consensus.cc:504] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:37.618536 1548 leader_election.cc:288] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:37.619045 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e03e76474584273ae050595a2286559" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:37.619156 32683 raft_consensus.cc:3037] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.620148 32683 raft_consensus.cc:2454] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:37.620651 32487 leader_election.cc:302] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:37.620715 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e03e76474584273ae050595a2286559" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:37.620812 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e03e76474584273ae050595a2286559" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:37.620859 32416 raft_consensus.cc:2454] T 1e03e76474584273ae050595a2286559 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:37.620935 32415 raft_consensus.cc:3037] T 1e03e76474584273ae050595a2286559 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.621863 32415 raft_consensus.cc:2454] T 1e03e76474584273ae050595a2286559 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:37.622452 1548 raft_consensus.cc:2781] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:37.622545 1548 raft_consensus.cc:686] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:09:37.622684 1548 consensus_queue.cc:237] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:37.624362 29867 catalog_manager.cc:5484] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:37.636790 32534 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:37.644362 30256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:37.648747 32400 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:37.649195 29993 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:37.650707 30387 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:37.652314 29864 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:37.656272 30125 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:37.664620 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 32517482e5bf43298ef858d5045381bd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1ff1205e1d604599812915480726501a]), partition=
I20240501 14:09:37.664775 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 32517482e5bf43298ef858d5045381bd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1ff1205e1d604599812915480726501a]), partition=
I20240501 14:09:37.664919 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32517482e5bf43298ef858d5045381bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.665042 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32517482e5bf43298ef858d5045381bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.666882 2188 tablet_bootstrap.cc:492] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:37.667105 2192 tablet_bootstrap.cc:492] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:37.667101 328 tablet_service.cc:1467] Processing CreateTablet for tablet 32517482e5bf43298ef858d5045381bd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1ff1205e1d604599812915480726501a]), partition=
I20240501 14:09:37.667344 328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32517482e5bf43298ef858d5045381bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.667863 2188 tablet_bootstrap.cc:654] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.667965 2192 tablet_bootstrap.cc:654] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.669687 2190 tablet_bootstrap.cc:492] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:37.670564 2190 tablet_bootstrap.cc:654] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.672708 328 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:37.678826 2190 tablet_bootstrap.cc:492] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:37.678921 2190 ts_tablet_manager.cc:1367] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 14:09:37.679396 2190 raft_consensus.cc:348] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:37.679500 2190 raft_consensus.cc:374] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.679529 2190 raft_consensus.cc:729] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.679632 2190 consensus_queue.cc:260] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:37.679881 2190 ts_tablet_manager.cc:1398] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.681139 2192 tablet_bootstrap.cc:492] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:37.681238 2192 ts_tablet_manager.cc:1367] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:09:37.681730 2192 raft_consensus.cc:348] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:37.681864 2192 raft_consensus.cc:374] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.681900 2192 raft_consensus.cc:729] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.682628 2192 consensus_queue.cc:260] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:37.682910 2192 ts_tablet_manager.cc:1398] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:37.691417 2188 tablet_bootstrap.cc:492] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:37.691540 2188 ts_tablet_manager.cc:1367] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240501 14:09:37.692036 2188 raft_consensus.cc:348] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:37.692157 2188 raft_consensus.cc:374] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.692188 2188 raft_consensus.cc:729] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.692342 2188 consensus_queue.cc:260] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:37.692581 2188 ts_tablet_manager.cc:1398] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.695089 32665 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:37.733834 30264 consensus_queue.cc:237] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:37.735267 32415 raft_consensus.cc:1261] T 1698ca3ec26741bb9b3131fe67c9682b P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.735267 30406 raft_consensus.cc:1261] T 1698ca3ec26741bb9b3131fe67c9682b P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.735759 2269 consensus_queue.cc:1035] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.735962 1945 consensus_queue.cc:1035] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.737848 2272 raft_consensus.cc:2932] T 1698ca3ec26741bb9b3131fe67c9682b P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:37.738278 32415 raft_consensus.cc:2932] T 1698ca3ec26741bb9b3131fe67c9682b P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:37.738283 30406 raft_consensus.cc:2932] T 1698ca3ec26741bb9b3131fe67c9682b P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:37.741370 29867 catalog_manager.cc:5484] T 1698ca3ec26741bb9b3131fe67c9682b P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:37.744704 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 1698ca3ec26741bb9b3131fe67c9682b with delete_type TABLET_DATA_TOMBSTONED (TS d35c16dcb506426993811bb2519d7db3 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:37.744956 2125 tablet_replica.cc:330] T 1698ca3ec26741bb9b3131fe67c9682b P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:37.745117 2125 raft_consensus.cc:2229] T 1698ca3ec26741bb9b3131fe67c9682b P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.745252 2125 raft_consensus.cc:2258] T 1698ca3ec26741bb9b3131fe67c9682b P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.745703 2125 ts_tablet_manager.cc:1875] T 1698ca3ec26741bb9b3131fe67c9682b P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:37.747776 2125 ts_tablet_manager.cc:1888] T 1698ca3ec26741bb9b3131fe67c9682b P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:37.747840 2125 log.cc:1196] T 1698ca3ec26741bb9b3131fe67c9682b P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/1698ca3ec26741bb9b3131fe67c9682b
I20240501 14:09:37.748283 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 1698ca3ec26741bb9b3131fe67c9682b (table rebalance_test_table_13 [id=422856ed458a4c52950e8e3e49019a55]) successfully deleted
I20240501 14:09:37.864791 2203 consensus_queue.cc:1035] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.868103 640 consensus_queue.cc:1035] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.903084 1253 consensus_queue.cc:1035] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.906541 562 consensus_queue.cc:1035] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.941742 2008 raft_consensus.cc:482] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:37.941890 2008 raft_consensus.cc:504] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:37.942463 2008 leader_election.cc:288] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:37.942890 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32517482e5bf43298ef858d5045381bd" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:37.942971 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32517482e5bf43298ef858d5045381bd" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:37.943080 32553 raft_consensus.cc:2454] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:37.943094 348 raft_consensus.cc:2454] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:37.943403 32621 leader_election.cc:302] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:37.943578 2008 raft_consensus.cc:2781] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:37.943625 2008 raft_consensus.cc:482] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:37.943673 2008 raft_consensus.cc:3037] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.944607 2008 raft_consensus.cc:504] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:37.944936 2008 leader_election.cc:288] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:37.945307 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32517482e5bf43298ef858d5045381bd" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:37.945436 348 raft_consensus.cc:3037] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.945444 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32517482e5bf43298ef858d5045381bd" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:37.945550 32553 raft_consensus.cc:3037] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.946480 348 raft_consensus.cc:2454] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:37.946544 32553 raft_consensus.cc:2454] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:37.946877 32621 leader_election.cc:302] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:37.947250 2008 raft_consensus.cc:2781] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:37.947381 2008 raft_consensus.cc:686] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Becoming Leader. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
I20240501 14:09:37.947495 2008 consensus_queue.cc:237] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:37.949286 29868 catalog_manager.cc:5484] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db935501e9104de5b15c8780681d6c38 (127.22.252.135). New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:37.979745 29993 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:37.983335 32534 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:37.987321 328 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:37.988762 30125 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:37.997009 30256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:38.003232 30387 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:38.013072 32665 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:38.024487 32400 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:38.029875 1548 consensus_queue.cc:1035] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.045207 1548 consensus_queue.cc:1035] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.056257 29868 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:38.061012 348 raft_consensus.cc:1261] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.061007 32553 raft_consensus.cc:1261] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.061573 2008 consensus_queue.cc:1035] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.061908 2008 consensus_queue.cc:1035] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.064939 29867 catalog_manager.cc:5576] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937) does not have the latest schema for tablet 32517482e5bf43298ef858d5045381bd (table rebalancer_dynamic_table [id=1ff1205e1d604599812915480726501a]). Expected version 1 got 0
I20240501 14:09:38.065294 2350 tablet.cc:1722] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:38.065605 2350 tablet.cc:1620] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:38.065742 29872 catalog_manager.cc:5576] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497) does not have the latest schema for tablet 32517482e5bf43298ef858d5045381bd (table rebalancer_dynamic_table [id=1ff1205e1d604599812915480726501a]). Expected version 1 got 0
I20240501 14:09:38.066715 2349 tablet.cc:1722] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:38.067085 2349 tablet.cc:1620] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:38.069360 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=1ff1205e1d604599812915480726501a] alter complete (version 1)
I20240501 14:09:38.073033 2351 tablet.cc:1722] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:38.073415 2351 tablet.cc:1620] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:38.073609 29872 catalog_manager.cc:5576] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497) does not have the latest schema for tablet 32517482e5bf43298ef858d5045381bd (table rebalancer_dynamic_table [id=1ff1205e1d604599812915480726501a]). Expected version 1 got 0
I20240501 14:09:38.078912 30013 consensus_queue.cc:237] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:38.080355 32415 raft_consensus.cc:1261] T fb36e198bcf34ef68a9595d7cbe93e80 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:38.080369 30264 raft_consensus.cc:1261] T fb36e198bcf34ef68a9595d7cbe93e80 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:38.080745 2358 consensus_queue.cc:1035] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:38.080919 1996 consensus_queue.cc:1035] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:38.082808 2361 raft_consensus.cc:2932] T fb36e198bcf34ef68a9595d7cbe93e80 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:38.083178 32415 raft_consensus.cc:2932] T fb36e198bcf34ef68a9595d7cbe93e80 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:38.083228 30264 raft_consensus.cc:2932] T fb36e198bcf34ef68a9595d7cbe93e80 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:38.085611 29872 catalog_manager.cc:5484] T fb36e198bcf34ef68a9595d7cbe93e80 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:38.088303 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet fb36e198bcf34ef68a9595d7cbe93e80 with delete_type TABLET_DATA_TOMBSTONED (TS d35c16dcb506426993811bb2519d7db3 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:38.088541 2125 tablet_replica.cc:330] T fb36e198bcf34ef68a9595d7cbe93e80 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:38.088688 2125 raft_consensus.cc:2229] T fb36e198bcf34ef68a9595d7cbe93e80 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:38.088869 2125 raft_consensus.cc:2258] T fb36e198bcf34ef68a9595d7cbe93e80 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:38.089452 2125 ts_tablet_manager.cc:1875] T fb36e198bcf34ef68a9595d7cbe93e80 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:38.091167 2125 ts_tablet_manager.cc:1888] T fb36e198bcf34ef68a9595d7cbe93e80 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:38.091217 2125 log.cc:1196] T fb36e198bcf34ef68a9595d7cbe93e80 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/fb36e198bcf34ef68a9595d7cbe93e80
I20240501 14:09:38.091737 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet fb36e198bcf34ef68a9595d7cbe93e80 (table rebalance_test_table_1 [id=df055757a36d4b9f89886608b024f0b8]) successfully deleted
I20240501 14:09:38.147980 29864 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:38.154448 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 0cccc8eec30141c1946b57d3be27000f (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=c6639c36fcc54bb09704a0d7c258496b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.154584 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 0cccc8eec30141c1946b57d3be27000f (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=c6639c36fcc54bb09704a0d7c258496b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.154764 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cccc8eec30141c1946b57d3be27000f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.154901 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cccc8eec30141c1946b57d3be27000f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.157063 2192 tablet_bootstrap.cc:492] T 0cccc8eec30141c1946b57d3be27000f P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:38.157146 2188 tablet_bootstrap.cc:492] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:38.158010 2188 tablet_bootstrap.cc:654] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.158015 2192 tablet_bootstrap.cc:654] T 0cccc8eec30141c1946b57d3be27000f P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.159289 32534 tablet_service.cc:1467] Processing CreateTablet for tablet caebf782fa60459b9e298aa9f1074449 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=c6639c36fcc54bb09704a0d7c258496b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.159291 30256 tablet_service.cc:1467] Processing CreateTablet for tablet 8c579bea324147af88ea88ac30153f02 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=c6639c36fcc54bb09704a0d7c258496b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.159375 329 tablet_service.cc:1467] Processing CreateTablet for tablet caebf782fa60459b9e298aa9f1074449 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=c6639c36fcc54bb09704a0d7c258496b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.159518 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caebf782fa60459b9e298aa9f1074449. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.159525 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c579bea324147af88ea88ac30153f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.159616 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caebf782fa60459b9e298aa9f1074449. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.161317 2188 tablet_bootstrap.cc:492] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:38.161408 2188 ts_tablet_manager.cc:1367] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:38.161782 2190 tablet_bootstrap.cc:492] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:38.161911 328 tablet_service.cc:1467] Processing CreateTablet for tablet 0cccc8eec30141c1946b57d3be27000f (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=c6639c36fcc54bb09704a0d7c258496b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.161898 2188 raft_consensus.cc:348] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:38.162005 2188 raft_consensus.cc:374] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.162039 2188 raft_consensus.cc:729] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.162150 328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cccc8eec30141c1946b57d3be27000f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.162158 2188 consensus_queue.cc:260] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:38.162474 2188 ts_tablet_manager.cc:1398] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.162955 2190 tablet_bootstrap.cc:654] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.164717 2192 tablet_bootstrap.cc:492] T 0cccc8eec30141c1946b57d3be27000f P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:38.164803 2192 ts_tablet_manager.cc:1367] T 0cccc8eec30141c1946b57d3be27000f P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:38.165282 2192 raft_consensus.cc:348] T 0cccc8eec30141c1946b57d3be27000f P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:38.165386 2192 raft_consensus.cc:374] T 0cccc8eec30141c1946b57d3be27000f P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.165418 2192 raft_consensus.cc:729] T 0cccc8eec30141c1946b57d3be27000f P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.165524 2192 consensus_queue.cc:260] T 0cccc8eec30141c1946b57d3be27000f P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:38.165812 2192 ts_tablet_manager.cc:1398] T 0cccc8eec30141c1946b57d3be27000f P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.166577 2192 tablet_bootstrap.cc:492] T caebf782fa60459b9e298aa9f1074449 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:38.166682 2398 tablet_bootstrap.cc:492] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:38.167474 2190 tablet_bootstrap.cc:492] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:38.167562 2190 ts_tablet_manager.cc:1367] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:38.167718 2192 tablet_bootstrap.cc:654] T caebf782fa60459b9e298aa9f1074449 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.167920 32665 tablet_service.cc:1467] Processing CreateTablet for tablet caebf782fa60459b9e298aa9f1074449 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=c6639c36fcc54bb09704a0d7c258496b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.168041 2190 raft_consensus.cc:348] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:38.168133 2190 raft_consensus.cc:374] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.168136 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caebf782fa60459b9e298aa9f1074449. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.168161 2190 raft_consensus.cc:729] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.168339 2190 consensus_queue.cc:260] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:38.168612 2190 ts_tablet_manager.cc:1398] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.168790 2190 tablet_bootstrap.cc:492] T 0cccc8eec30141c1946b57d3be27000f P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:38.169652 2190 tablet_bootstrap.cc:654] T 0cccc8eec30141c1946b57d3be27000f P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.170508 30387 tablet_service.cc:1467] Processing CreateTablet for tablet 8c579bea324147af88ea88ac30153f02 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=c6639c36fcc54bb09704a0d7c258496b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.170785 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c579bea324147af88ea88ac30153f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.171409 29872 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:38.173136 2188 tablet_bootstrap.cc:492] T caebf782fa60459b9e298aa9f1074449 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:38.173359 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 8c579bea324147af88ea88ac30153f02 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=c6639c36fcc54bb09704a0d7c258496b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.173579 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c579bea324147af88ea88ac30153f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.174063 2188 tablet_bootstrap.cc:654] T caebf782fa60459b9e298aa9f1074449 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.175153 2192 tablet_bootstrap.cc:492] T caebf782fa60459b9e298aa9f1074449 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:38.175232 2192 ts_tablet_manager.cc:1367] T caebf782fa60459b9e298aa9f1074449 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 14:09:38.175624 2192 raft_consensus.cc:348] T caebf782fa60459b9e298aa9f1074449 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:38.175722 2192 raft_consensus.cc:374] T caebf782fa60459b9e298aa9f1074449 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.175750 2192 raft_consensus.cc:729] T caebf782fa60459b9e298aa9f1074449 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.175859 2192 consensus_queue.cc:260] T caebf782fa60459b9e298aa9f1074449 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:38.176082 2192 ts_tablet_manager.cc:1398] T caebf782fa60459b9e298aa9f1074449 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.180125 2190 tablet_bootstrap.cc:492] T 0cccc8eec30141c1946b57d3be27000f P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:38.180292 2190 ts_tablet_manager.cc:1367] T 0cccc8eec30141c1946b57d3be27000f P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:38.180876 2190 raft_consensus.cc:348] T 0cccc8eec30141c1946b57d3be27000f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:38.181221 2190 raft_consensus.cc:374] T 0cccc8eec30141c1946b57d3be27000f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.182240 2398 tablet_bootstrap.cc:654] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.182821 2402 tablet_bootstrap.cc:492] T 8c579bea324147af88ea88ac30153f02 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:38.183694 2402 tablet_bootstrap.cc:654] T 8c579bea324147af88ea88ac30153f02 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.184589 2349 tablet.cc:1722] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:38.184845 2349 tablet.cc:1620] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:38.185361 2351 tablet.cc:1722] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:38.185612 2351 tablet.cc:1620] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:38.187602 2398 tablet_bootstrap.cc:492] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:38.187698 2398 ts_tablet_manager.cc:1367] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240501 14:09:38.188187 2398 raft_consensus.cc:348] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:38.189154 2398 raft_consensus.cc:374] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.189263 2398 raft_consensus.cc:729] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.189445 2398 consensus_queue.cc:260] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:38.190979 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=1ff1205e1d604599812915480726501a] alter complete (version 2)
I20240501 14:09:38.186173 2350 tablet.cc:1722] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:38.191876 2350 tablet.cc:1620] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:38.192164 2402 tablet_bootstrap.cc:492] T 8c579bea324147af88ea88ac30153f02 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:38.192278 2402 ts_tablet_manager.cc:1367] T 8c579bea324147af88ea88ac30153f02 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:38.181311 2190 raft_consensus.cc:729] T 0cccc8eec30141c1946b57d3be27000f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.193738 2190 consensus_queue.cc:260] T 0cccc8eec30141c1946b57d3be27000f P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:38.192673 2402 raft_consensus.cc:348] T 8c579bea324147af88ea88ac30153f02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:38.194002 2402 raft_consensus.cc:374] T 8c579bea324147af88ea88ac30153f02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.194087 2402 raft_consensus.cc:729] T 8c579bea324147af88ea88ac30153f02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.194209 2190 ts_tablet_manager.cc:1398] T 0cccc8eec30141c1946b57d3be27000f P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240501 14:09:38.194270 2402 consensus_queue.cc:260] T 8c579bea324147af88ea88ac30153f02 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:38.194682 2402 ts_tablet_manager.cc:1398] T 8c579bea324147af88ea88ac30153f02 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:38.198043 2008 raft_consensus.cc:482] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:38.198130 2008 raft_consensus.cc:504] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:38.198503 2008 leader_election.cc:288] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:38.199400 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cccc8eec30141c1946b57d3be27000f" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:38.199512 348 raft_consensus.cc:2454] T 0cccc8eec30141c1946b57d3be27000f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:38.199752 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cccc8eec30141c1946b57d3be27000f" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:38.199862 32553 raft_consensus.cc:2454] T 0cccc8eec30141c1946b57d3be27000f P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:38.199904 32620 leader_election.cc:302] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:38.200227 2008 raft_consensus.cc:2781] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:38.200322 2008 raft_consensus.cc:482] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:38.200361 2008 raft_consensus.cc:3037] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.200723 2188 tablet_bootstrap.cc:492] T caebf782fa60459b9e298aa9f1074449 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:38.200804 2188 ts_tablet_manager.cc:1367] T caebf782fa60459b9e298aa9f1074449 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.000s
I20240501 14:09:38.201261 2188 raft_consensus.cc:348] T caebf782fa60459b9e298aa9f1074449 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:38.201303 2008 raft_consensus.cc:504] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:38.201650 2008 leader_election.cc:288] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:38.202032 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cccc8eec30141c1946b57d3be27000f" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:38.202126 32553 raft_consensus.cc:3037] T 0cccc8eec30141c1946b57d3be27000f P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.202523 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cccc8eec30141c1946b57d3be27000f" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:38.202639 348 raft_consensus.cc:3037] T 0cccc8eec30141c1946b57d3be27000f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.202980 32553 raft_consensus.cc:2454] T 0cccc8eec30141c1946b57d3be27000f P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:38.203361 32621 leader_election.cc:302] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:38.203619 348 raft_consensus.cc:2454] T 0cccc8eec30141c1946b57d3be27000f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:38.203755 2008 raft_consensus.cc:2781] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:38.203827 2008 raft_consensus.cc:686] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Becoming Leader. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
I20240501 14:09:38.203946 2008 consensus_queue.cc:237] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:38.201354 2188 raft_consensus.cc:374] T caebf782fa60459b9e298aa9f1074449 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.205493 2188 raft_consensus.cc:729] T caebf782fa60459b9e298aa9f1074449 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.205615 2188 consensus_queue.cc:260] T caebf782fa60459b9e298aa9f1074449 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:38.205924 2188 ts_tablet_manager.cc:1398] T caebf782fa60459b9e298aa9f1074449 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:38.206074 2188 tablet_bootstrap.cc:492] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:38.207036 2188 tablet_bootstrap.cc:654] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.207037 2398 ts_tablet_manager.cc:1398] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.019s user 0.001s sys 0.000s
I20240501 14:09:38.209092 2188 tablet_bootstrap.cc:492] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:38.209170 2188 ts_tablet_manager.cc:1367] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:38.209602 2188 raft_consensus.cc:348] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:38.209712 2188 raft_consensus.cc:374] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.209745 2188 raft_consensus.cc:729] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.209848 2188 consensus_queue.cc:260] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:38.210127 2188 ts_tablet_manager.cc:1398] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:38.210783 29872 catalog_manager.cc:5484] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db935501e9104de5b15c8780681d6c38 (127.22.252.135). New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:38.225078 32534 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:38.227061 30256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:38.231909 32400 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:38.231910 30125 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:38.235291 328 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:38.243382 29993 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:38.247651 32664 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:38.279146 30387 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:38.295545 29872 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:38.301473 2321 raft_consensus.cc:482] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:38.301611 2321 raft_consensus.cc:504] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:38.302178 2321 leader_election.cc:288] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:38.302311 2349 tablet.cc:1722] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:38.303030 2351 tablet.cc:1722] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:38.306232 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c579bea324147af88ea88ac30153f02" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:38.306367 30406 raft_consensus.cc:2454] T 8c579bea324147af88ea88ac30153f02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:38.306710 32618 leader_election.cc:302] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:38.307073 2321 raft_consensus.cc:2781] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:38.307124 2321 raft_consensus.cc:482] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:38.307159 2321 raft_consensus.cc:3037] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.308935 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c579bea324147af88ea88ac30153f02" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:38.309065 30264 raft_consensus.cc:2454] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:38.309985 2350 tablet.cc:1722] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:38.310046 29868 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=1ff1205e1d604599812915480726501a] alter complete (version 3)
I20240501 14:09:38.308178 2321 raft_consensus.cc:504] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:38.313143 2321 leader_election.cc:288] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:38.313829 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c579bea324147af88ea88ac30153f02" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:38.313941 30264 raft_consensus.cc:3037] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.314719 30264 raft_consensus.cc:2454] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:38.315203 32620 leader_election.cc:302] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:38.315367 2321 raft_consensus.cc:2781] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:38.315449 2321 raft_consensus.cc:686] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Becoming Leader. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
I20240501 14:09:38.315438 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c579bea324147af88ea88ac30153f02" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:38.315567 30406 raft_consensus.cc:3037] T 8c579bea324147af88ea88ac30153f02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.315565 2321 consensus_queue.cc:237] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:38.316495 30406 raft_consensus.cc:2454] T 8c579bea324147af88ea88ac30153f02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:38.317370 29865 catalog_manager.cc:5484] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db935501e9104de5b15c8780681d6c38 (127.22.252.135). New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:38.336138 30264 consensus_queue.cc:237] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.337687 30406 raft_consensus.cc:1261] T c5d406c64b2e432a88e8c675b91dd144 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:38.337702 30013 raft_consensus.cc:1261] T c5d406c64b2e432a88e8c675b91dd144 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:38.338071 2441 consensus_queue.cc:1035] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.338555 2440 consensus_queue.cc:1035] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.340436 2440 raft_consensus.cc:2932] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:38.340859 30406 raft_consensus.cc:2932] T c5d406c64b2e432a88e8c675b91dd144 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
W20240501 14:09:38.341490 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5d406c64b2e432a88e8c675b91dd144. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.342460 30013 raft_consensus.cc:2932] T c5d406c64b2e432a88e8c675b91dd144 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:38.343796 29868 catalog_manager.cc:5484] T c5d406c64b2e432a88e8c675b91dd144 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:38.357021 640 raft_consensus.cc:482] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:38.357132 640 raft_consensus.cc:504] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:38.357522 640 leader_election.cc:288] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:38.357882 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caebf782fa60459b9e298aa9f1074449" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:38.357910 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caebf782fa60459b9e298aa9f1074449" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:38.358031 32553 raft_consensus.cc:2454] T caebf782fa60459b9e298aa9f1074449 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:38.358047 32683 raft_consensus.cc:2454] T caebf782fa60459b9e298aa9f1074449 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:38.358333 32750 leader_election.cc:302] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:38.358510 640 raft_consensus.cc:2781] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:38.358574 640 raft_consensus.cc:482] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:38.358616 640 raft_consensus.cc:3037] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.359630 640 raft_consensus.cc:504] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:38.359974 640 leader_election.cc:288] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:38.360323 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caebf782fa60459b9e298aa9f1074449" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:38.360335 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caebf782fa60459b9e298aa9f1074449" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:38.360420 32683 raft_consensus.cc:3037] T caebf782fa60459b9e298aa9f1074449 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.360442 32553 raft_consensus.cc:3037] T caebf782fa60459b9e298aa9f1074449 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.361191 32683 raft_consensus.cc:2454] T caebf782fa60459b9e298aa9f1074449 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:38.361299 32553 raft_consensus.cc:2454] T caebf782fa60459b9e298aa9f1074449 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:38.361506 32750 leader_election.cc:302] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:38.362037 640 raft_consensus.cc:2781] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:38.362123 640 raft_consensus.cc:686] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:38.362274 640 consensus_queue.cc:237] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:38.362932 30264 consensus_queue.cc:237] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.364003 29872 catalog_manager.cc:5484] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:38.364295 30406 raft_consensus.cc:1261] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:38.364638 2441 consensus_queue.cc:1035] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.365020 30146 raft_consensus.cc:1261] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:38.365263 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44949ddf1eac45cfa4fc8e1c532cf7bb. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.365599 2441 consensus_queue.cc:1035] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.367262 2445 raft_consensus.cc:2932] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:38.367647 30146 raft_consensus.cc:2932] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:38.369591 30406 raft_consensus.cc:2932] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:38.370512 29872 catalog_manager.cc:5484] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d35c16dcb506426993811bb2519d7db3 reported cstate change: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:38.387145 348 consensus_queue.cc:237] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:38.388412 32415 raft_consensus.cc:1261] T f2541122e05f45c2843d06c5342c8d01 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.388479 30013 raft_consensus.cc:1261] T f2541122e05f45c2843d06c5342c8d01 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.388751 2481 consensus_queue.cc:1035] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:38.388900 640 consensus_queue.cc:1035] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:09:38.389082 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2541122e05f45c2843d06c5342c8d01. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.390784 640 raft_consensus.cc:2932] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:38.391191 30013 raft_consensus.cc:2932] T f2541122e05f45c2843d06c5342c8d01 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:38.392570 32415 raft_consensus.cc:2932] T f2541122e05f45c2843d06c5342c8d01 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:38.393584 29869 catalog_manager.cc:5484] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: config changed from index -1 to 2, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:38.412115 32415 consensus_queue.cc:237] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:38.412585 29872 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:38.412703 29872 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
W20240501 14:09:38.413913 32356 consensus_peers.cc:485] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 -> Peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): Couldn't send request to peer d1c217469ae4478ebd49e4ed9f7352d7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35dd2c9d436448c482ef0b3fac3a3b64. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.414433 348 raft_consensus.cc:1261] T 35dd2c9d436448c482ef0b3fac3a3b64 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.414657 29872 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 32517482e5bf43298ef858d5045381bd
I20240501 14:09:38.414906 32553 raft_consensus.cc:1261] T 35dd2c9d436448c482ef0b3fac3a3b64 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.415213 328 tablet_service.cc:1514] Processing DeleteTablet for tablet 32517482e5bf43298ef858d5045381bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:38 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:38.415377 32534 tablet_service.cc:1514] Processing DeleteTablet for tablet 32517482e5bf43298ef858d5045381bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:38 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:38.415462 2499 consensus_queue.cc:1035] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:38.415670 32664 tablet_service.cc:1514] Processing DeleteTablet for tablet 32517482e5bf43298ef858d5045381bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:38 UTC) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:38.415827 1253 consensus_queue.cc:1035] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:38.415968 2502 tablet_replica.cc:330] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:38.416095 2502 raft_consensus.cc:2229] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:38.416514 2502 raft_consensus.cc:2258] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:38.417171 2502 ts_tablet_manager.cc:1875] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:38.417410 2500 tablet_replica.cc:330] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:38.417675 2500 raft_consensus.cc:2229] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:38.417848 2500 raft_consensus.cc:2258] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:38.418270 1253 raft_consensus.cc:2932] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:38.419770 2502 ts_tablet_manager.cc:1888] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:38.419898 2502 log.cc:1196] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/32517482e5bf43298ef858d5045381bd
I20240501 14:09:38.420322 2500 ts_tablet_manager.cc:1875] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:38.420488 2502 ts_tablet_manager.cc:1909] T 32517482e5bf43298ef858d5045381bd P db935501e9104de5b15c8780681d6c38: Deleting consensus metadata
I20240501 14:09:38.420176 32553 raft_consensus.cc:2932] T 35dd2c9d436448c482ef0b3fac3a3b64 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:38.421198 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet 32517482e5bf43298ef858d5045381bd (table rebalancer_dynamic_table_new_name [id=1ff1205e1d604599812915480726501a]) successfully deleted
I20240501 14:09:38.422371 29872 catalog_manager.cc:5484] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:38.423691 2500 ts_tablet_manager.cc:1888] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:38.423807 2500 log.cc:1196] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/32517482e5bf43298ef858d5045381bd
I20240501 14:09:38.424124 2500 ts_tablet_manager.cc:1909] T 32517482e5bf43298ef858d5045381bd P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:09:38.425050 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 32517482e5bf43298ef858d5045381bd (table rebalancer_dynamic_table_new_name [id=1ff1205e1d604599812915480726501a]) successfully deleted
I20240501 14:09:38.426137 348 raft_consensus.cc:2932] T 35dd2c9d436448c482ef0b3fac3a3b64 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:38.428722 2501 tablet_replica.cc:330] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:38.429240 2501 raft_consensus.cc:2229] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:38.429406 2501 raft_consensus.cc:2258] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:38.429965 2501 ts_tablet_manager.cc:1875] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:38.433249 2501 ts_tablet_manager.cc:1888] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:38.433375 2501 log.cc:1196] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/32517482e5bf43298ef858d5045381bd
I20240501 14:09:38.433712 2501 ts_tablet_manager.cc:1909] T 32517482e5bf43298ef858d5045381bd P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:38.434389 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet 32517482e5bf43298ef858d5045381bd (table rebalancer_dynamic_table_new_name [id=1ff1205e1d604599812915480726501a]) successfully deleted
I20240501 14:09:38.446141 30264 consensus_queue.cc:237] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:38.447319 30013 raft_consensus.cc:1261] T 75b67ccb9a334b25ae71b553a2a84a02 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:38.447633 30406 raft_consensus.cc:1261] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:38.447697 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75b67ccb9a334b25ae71b553a2a84a02. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.447696 2272 consensus_queue.cc:1035] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.447988 2445 consensus_queue.cc:1035] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.449717 2272 raft_consensus.cc:2932] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:38.450138 30013 raft_consensus.cc:2932] T 75b67ccb9a334b25ae71b553a2a84a02 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:38.451967 30406 raft_consensus.cc:2932] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:38.452543 29865 catalog_manager.cc:5484] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:38.470136 348 consensus_queue.cc:237] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:38.471411 32415 raft_consensus.cc:1261] T 66471362a1d24a8dab53f9cad512a73a P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:38.471448 32750 consensus_peers.cc:485] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): Couldn't send request to peer d1c217469ae4478ebd49e4ed9f7352d7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66471362a1d24a8dab53f9cad512a73a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.471745 30013 raft_consensus.cc:1261] T 66471362a1d24a8dab53f9cad512a73a P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.471768 2203 consensus_queue.cc:1035] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:38.472071 640 consensus_queue.cc:1035] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:38.473753 2203 raft_consensus.cc:2932] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:38.474108 30013 raft_consensus.cc:2932] T 66471362a1d24a8dab53f9cad512a73a P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:38.476095 29869 catalog_manager.cc:5484] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:38.478257 32415 raft_consensus.cc:2932] T 66471362a1d24a8dab53f9cad512a73a P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:38.492653 30277 consensus_queue.cc:237] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.494078 30146 raft_consensus.cc:1261] T 0517ff3519644b489b912f8e5a36274c P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:38.494160 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0517ff3519644b489b912f8e5a36274c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.494410 30013 raft_consensus.cc:1261] T 0517ff3519644b489b912f8e5a36274c P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:38.494757 2445 consensus_queue.cc:1035] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.494915 2272 consensus_queue.cc:1035] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.496897 2272 raft_consensus.cc:2932] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:38.497351 30013 raft_consensus.cc:2932] T 0517ff3519644b489b912f8e5a36274c P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:38.500710 29869 catalog_manager.cc:5484] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:38.500950 30146 raft_consensus.cc:2932] T 0517ff3519644b489b912f8e5a36274c P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:38.517251 29872 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:38.522053 328 tablet_service.cc:1467] Processing CreateTablet for tablet 02ab742a1e42416da3203b289ae3ba36 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=30dffeac8b584dedb2b82736abfdc127]), partition=
I20240501 14:09:38.522286 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 02ab742a1e42416da3203b289ae3ba36 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=30dffeac8b584dedb2b82736abfdc127]), partition=
I20240501 14:09:38.522328 328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ab742a1e42416da3203b289ae3ba36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.522519 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ab742a1e42416da3203b289ae3ba36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.524010 2190 tablet_bootstrap.cc:492] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:38.524222 2192 tablet_bootstrap.cc:492] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:38.524325 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 02ab742a1e42416da3203b289ae3ba36 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=30dffeac8b584dedb2b82736abfdc127]), partition=
I20240501 14:09:38.524556 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ab742a1e42416da3203b289ae3ba36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.524968 2190 tablet_bootstrap.cc:654] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.525292 2192 tablet_bootstrap.cc:654] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.526397 2190 tablet_bootstrap.cc:492] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:38.526490 2190 ts_tablet_manager.cc:1367] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:38.526933 2190 raft_consensus.cc:348] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.527038 2190 raft_consensus.cc:374] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.527067 2190 raft_consensus.cc:729] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.527264 2190 consensus_queue.cc:260] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.527513 2190 ts_tablet_manager.cc:1398] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.528188 2192 tablet_bootstrap.cc:492] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:38.528368 2192 ts_tablet_manager.cc:1367] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:09:38.528445 2571 tablet_bootstrap.cc:492] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:38.528810 2192 raft_consensus.cc:348] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.528930 2192 raft_consensus.cc:374] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.528965 2192 raft_consensus.cc:729] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.529072 2192 consensus_queue.cc:260] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.529507 2571 tablet_bootstrap.cc:654] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.529930 2192 ts_tablet_manager.cc:1398] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:38.531023 2571 tablet_bootstrap.cc:492] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:38.531178 2571 ts_tablet_manager.cc:1367] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:38.531631 2571 raft_consensus.cc:348] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.531780 2571 raft_consensus.cc:374] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.531883 2571 raft_consensus.cc:729] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.532095 2571 consensus_queue.cc:260] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.533170 2571 ts_tablet_manager.cc:1398] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:38.624763 640 raft_consensus.cc:482] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:38.624871 640 raft_consensus.cc:504] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.625248 640 leader_election.cc:288] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:38.625552 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ab742a1e42416da3203b289ae3ba36" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:38.625675 32415 raft_consensus.cc:2454] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:38.625658 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ab742a1e42416da3203b289ae3ba36" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:38.625784 32553 raft_consensus.cc:2454] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:38.625995 32752 leader_election.cc:302] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:38.626189 640 raft_consensus.cc:2781] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:38.626241 640 raft_consensus.cc:482] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:38.626268 640 raft_consensus.cc:3037] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.627048 640 raft_consensus.cc:504] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.627364 640 leader_election.cc:288] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:38.627651 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ab742a1e42416da3203b289ae3ba36" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:38.627749 32415 raft_consensus.cc:3037] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.627748 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ab742a1e42416da3203b289ae3ba36" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:38.627849 32553 raft_consensus.cc:3037] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.628682 32553 raft_consensus.cc:2454] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:38.628923 32415 raft_consensus.cc:2454] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:38.628938 32753 leader_election.cc:302] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:38.629092 640 raft_consensus.cc:2781] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:38.629175 640 raft_consensus.cc:686] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:38.629321 640 consensus_queue.cc:237] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.631181 29869 catalog_manager.cc:5484] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:38.656090 30256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:38.660037 32534 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:38.661808 30387 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:38.664486 2421 consensus_queue.cc:1035] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.666486 32400 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:38.668488 29993 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:38.668565 328 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:38.677125 2321 consensus_queue.cc:1035] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.694854 32664 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:38.707566 30125 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:38.740453 2008 consensus_queue.cc:1035] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.743463 2421 consensus_queue.cc:1035] T 8c579bea324147af88ea88ac30153f02 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.751566 29865 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:38.755421 32415 raft_consensus.cc:1261] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.755796 32553 raft_consensus.cc:1261] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.756196 640 consensus_queue.cc:1035] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.756871 2481 consensus_queue.cc:1035] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.759450 29869 catalog_manager.cc:5576] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497) does not have the latest schema for tablet 02ab742a1e42416da3203b289ae3ba36 (table rebalancer_dynamic_table [id=30dffeac8b584dedb2b82736abfdc127]). Expected version 1 got 0
I20240501 14:09:38.759454 2350 tablet.cc:1722] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:38.759799 2350 tablet.cc:1620] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:38.760318 29865 catalog_manager.cc:5576] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265) does not have the latest schema for tablet 02ab742a1e42416da3203b289ae3ba36 (table rebalancer_dynamic_table [id=30dffeac8b584dedb2b82736abfdc127]). Expected version 1 got 0
I20240501 14:09:38.763212 2351 tablet.cc:1722] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:38.763486 2351 tablet.cc:1620] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:38.764226 29869 catalog_manager.cc:5576] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265) does not have the latest schema for tablet 02ab742a1e42416da3203b289ae3ba36 (table rebalancer_dynamic_table [id=30dffeac8b584dedb2b82736abfdc127]). Expected version 1 got 0
I20240501 14:09:38.765225 29865 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=30dffeac8b584dedb2b82736abfdc127] alter complete (version 1)
I20240501 14:09:38.767374 2643 tablet.cc:1722] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:38.767627 2643 tablet.cc:1620] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:38.773523 640 consensus_queue.cc:1035] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.776912 2203 consensus_queue.cc:1035] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.834477 30256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:38.839411 30387 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:38.840119 29993 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:38.840569 30125 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:38.842309 32534 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:38.849543 32400 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:38.855202 32664 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:38.870533 29864 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:38.871412 2678 ts_tablet_manager.cc:902] T c5d406c64b2e432a88e8c675b91dd144 P 5f8a0cfdfdca4148a75686dc88059cf3: Initiating tablet copy from peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:38.871912 2678 tablet_copy_client.cc:308] T c5d406c64b2e432a88e8c675b91dd144 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.131:43103
I20240501 14:09:38.876164 29868 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:38.879207 2350 tablet.cc:1722] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:38.879503 2350 tablet.cc:1620] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:38.880317 2351 tablet.cc:1722] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:38.880575 2351 tablet.cc:1620] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:38.888962 2643 tablet.cc:1722] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:38.889258 2643 tablet.cc:1620] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:38.890194 29866 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=30dffeac8b584dedb2b82736abfdc127] alter complete (version 2)
I20240501 14:09:38.895226 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 65bc0e6ad1fa491e8e5ae225f86561f9 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=6139b5f2fcc94630ac7d807b61ccd1ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.895475 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65bc0e6ad1fa491e8e5ae225f86561f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.896397 32400 tablet_service.cc:1467] Processing CreateTablet for tablet b0ef35582c4f4d5c95199e00ccac92c9 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=6139b5f2fcc94630ac7d807b61ccd1ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.896402 30387 tablet_service.cc:1467] Processing CreateTablet for tablet b0ef35582c4f4d5c95199e00ccac92c9 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=6139b5f2fcc94630ac7d807b61ccd1ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.896631 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ef35582c4f4d5c95199e00ccac92c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.896631 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ef35582c4f4d5c95199e00ccac92c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.897716 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 65bc0e6ad1fa491e8e5ae225f86561f9 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=6139b5f2fcc94630ac7d807b61ccd1ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.897943 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65bc0e6ad1fa491e8e5ae225f86561f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.898609 2571 tablet_bootstrap.cc:492] T b0ef35582c4f4d5c95199e00ccac92c9 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:38.899260 328 tablet_service.cc:1467] Processing CreateTablet for tablet 65bc0e6ad1fa491e8e5ae225f86561f9 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=6139b5f2fcc94630ac7d807b61ccd1ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.899523 328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65bc0e6ad1fa491e8e5ae225f86561f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.899574 2571 tablet_bootstrap.cc:654] T b0ef35582c4f4d5c95199e00ccac92c9 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.900153 2192 tablet_bootstrap.cc:492] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:38.900626 327 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:38.900998 2192 tablet_bootstrap.cc:654] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.901753 329 tablet_service.cc:1467] Processing CreateTablet for tablet 7cb8f294e9704c549e1e0590b1429fe5 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=6139b5f2fcc94630ac7d807b61ccd1ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.901847 2571 tablet_bootstrap.cc:492] T b0ef35582c4f4d5c95199e00ccac92c9 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:38.901918 2571 ts_tablet_manager.cc:1367] T b0ef35582c4f4d5c95199e00ccac92c9 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:38.901980 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cb8f294e9704c549e1e0590b1429fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.902388 2684 tablet_bootstrap.cc:492] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:38.902393 2571 raft_consensus.cc:348] T b0ef35582c4f4d5c95199e00ccac92c9 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.902485 2571 raft_consensus.cc:374] T b0ef35582c4f4d5c95199e00ccac92c9 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.902511 2571 raft_consensus.cc:729] T b0ef35582c4f4d5c95199e00ccac92c9 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.902613 2571 consensus_queue.cc:260] T b0ef35582c4f4d5c95199e00ccac92c9 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.902874 2571 ts_tablet_manager.cc:1398] T b0ef35582c4f4d5c95199e00ccac92c9 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.903410 2685 tablet_bootstrap.cc:492] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:38.903642 2192 tablet_bootstrap.cc:492] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:38.903723 2192 ts_tablet_manager.cc:1367] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:09:38.901800 2190 tablet_bootstrap.cc:492] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:38.904150 2192 raft_consensus.cc:348] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:38.904286 2192 raft_consensus.cc:374] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.904309 2685 tablet_bootstrap.cc:654] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.904320 2192 raft_consensus.cc:729] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.904414 2192 consensus_queue.cc:260] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:38.904690 2192 ts_tablet_manager.cc:1398] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.905457 2684 tablet_bootstrap.cc:654] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.905464 30297 tablet_copy_service.cc:140] P 1d0bb69611af4876a2b3b33c4766e13a: Received BeginTabletCopySession request for tablet c5d406c64b2e432a88e8c675b91dd144 from peer 5f8a0cfdfdca4148a75686dc88059cf3 ({username='slave'} at 127.22.252.134:55213)
I20240501 14:09:38.905560 30297 tablet_copy_service.cc:161] P 1d0bb69611af4876a2b3b33c4766e13a: Beginning new tablet copy session on tablet c5d406c64b2e432a88e8c675b91dd144 from peer 5f8a0cfdfdca4148a75686dc88059cf3 at {username='slave'} at 127.22.252.134:55213: session id = 5f8a0cfdfdca4148a75686dc88059cf3-c5d406c64b2e432a88e8c675b91dd144
I20240501 14:09:38.906296 2190 tablet_bootstrap.cc:654] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.906435 32532 tablet_service.cc:1467] Processing CreateTablet for tablet 7cb8f294e9704c549e1e0590b1429fe5 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=6139b5f2fcc94630ac7d807b61ccd1ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.906673 32532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cb8f294e9704c549e1e0590b1429fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.906672 30297 tablet_copy_source_session.cc:205] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:38.907343 30125 tablet_service.cc:1467] Processing CreateTablet for tablet 7cb8f294e9704c549e1e0590b1429fe5 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=6139b5f2fcc94630ac7d807b61ccd1ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.907416 2678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d406c64b2e432a88e8c675b91dd144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.907631 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cb8f294e9704c549e1e0590b1429fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.909391 2678 tablet_copy_client.cc:783] T c5d406c64b2e432a88e8c675b91dd144 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:38.909518 2678 tablet_copy_client.cc:647] T c5d406c64b2e432a88e8c675b91dd144 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:38.910728 2192 tablet_bootstrap.cc:492] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:38.911675 32534 tablet_service.cc:1467] Processing CreateTablet for tablet b0ef35582c4f4d5c95199e00ccac92c9 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=6139b5f2fcc94630ac7d807b61ccd1ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.911897 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ef35582c4f4d5c95199e00ccac92c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.912578 2678 tablet_copy_client.cc:523] T c5d406c64b2e432a88e8c675b91dd144 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:38.913101 2192 tablet_bootstrap.cc:654] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.914016 2685 tablet_bootstrap.cc:492] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:38.914268 2685 ts_tablet_manager.cc:1367] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:09:38.914736 2685 raft_consensus.cc:348] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:38.914891 2685 raft_consensus.cc:374] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.914959 2685 raft_consensus.cc:729] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.915158 2685 consensus_queue.cc:260] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:38.915565 2190 tablet_bootstrap.cc:492] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:38.915575 2685 ts_tablet_manager.cc:1398] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.915652 2190 ts_tablet_manager.cc:1367] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20240501 14:09:38.916083 2678 tablet_bootstrap.cc:492] T c5d406c64b2e432a88e8c675b91dd144 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:38.916083 2190 raft_consensus.cc:348] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:38.916175 2190 raft_consensus.cc:374] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.916206 2190 raft_consensus.cc:729] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.916333 2190 consensus_queue.cc:260] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:38.916676 2190 ts_tablet_manager.cc:1398] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.916849 2190 tablet_bootstrap.cc:492] T 7cb8f294e9704c549e1e0590b1429fe5 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:38.916927 2684 tablet_bootstrap.cc:492] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:38.917048 2684 ts_tablet_manager.cc:1367] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 14:09:38.917735 2190 tablet_bootstrap.cc:654] T 7cb8f294e9704c549e1e0590b1429fe5 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.919113 2684 raft_consensus.cc:348] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.919303 2684 raft_consensus.cc:374] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.918864 2190 tablet_bootstrap.cc:492] T 7cb8f294e9704c549e1e0590b1429fe5 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:38.919389 2684 raft_consensus.cc:729] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.919469 2190 ts_tablet_manager.cc:1367] T 7cb8f294e9704c549e1e0590b1429fe5 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:38.919660 2684 consensus_queue.cc:260] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.920008 2684 ts_tablet_manager.cc:1398] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:38.920348 2691 tablet_bootstrap.cc:492] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:38.921308 2190 raft_consensus.cc:348] T 7cb8f294e9704c549e1e0590b1429fe5 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:38.921430 2190 raft_consensus.cc:374] T 7cb8f294e9704c549e1e0590b1429fe5 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.921468 2190 raft_consensus.cc:729] T 7cb8f294e9704c549e1e0590b1429fe5 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.921415 2691 tablet_bootstrap.cc:654] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.921615 2190 consensus_queue.cc:260] T 7cb8f294e9704c549e1e0590b1429fe5 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:38.922726 2192 tablet_bootstrap.cc:492] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:38.922808 2192 ts_tablet_manager.cc:1367] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:38.923032 2190 ts_tablet_manager.cc:1398] T 7cb8f294e9704c549e1e0590b1429fe5 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:38.923280 2192 raft_consensus.cc:348] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:38.923377 2192 raft_consensus.cc:374] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.923413 2192 raft_consensus.cc:729] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.923527 2192 consensus_queue.cc:260] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:38.923785 2192 ts_tablet_manager.cc:1398] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:38.923919 2192 tablet_bootstrap.cc:492] T b0ef35582c4f4d5c95199e00ccac92c9 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:38.925338 2192 tablet_bootstrap.cc:654] T b0ef35582c4f4d5c95199e00ccac92c9 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.926801 2696 ts_tablet_manager.cc:902] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 5f8a0cfdfdca4148a75686dc88059cf3: Initiating tablet copy from peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:38.927914 2192 tablet_bootstrap.cc:492] T b0ef35582c4f4d5c95199e00ccac92c9 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:38.928063 2192 ts_tablet_manager.cc:1367] T b0ef35582c4f4d5c95199e00ccac92c9 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:09:38.928565 2695 ts_tablet_manager.cc:902] T 0517ff3519644b489b912f8e5a36274c P 5f8a0cfdfdca4148a75686dc88059cf3: Initiating tablet copy from peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:38.928588 2192 raft_consensus.cc:348] T b0ef35582c4f4d5c95199e00ccac92c9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.928786 2192 raft_consensus.cc:374] T b0ef35582c4f4d5c95199e00ccac92c9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.928869 2192 raft_consensus.cc:729] T b0ef35582c4f4d5c95199e00ccac92c9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.929059 2192 consensus_queue.cc:260] T b0ef35582c4f4d5c95199e00ccac92c9 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.929399 2696 tablet_copy_client.cc:308] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.131:43103
I20240501 14:09:38.929674 2695 tablet_copy_client.cc:308] T 0517ff3519644b489b912f8e5a36274c P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.131:43103
I20240501 14:09:38.929867 30297 tablet_copy_service.cc:140] P 1d0bb69611af4876a2b3b33c4766e13a: Received BeginTabletCopySession request for tablet 44949ddf1eac45cfa4fc8e1c532cf7bb from peer 5f8a0cfdfdca4148a75686dc88059cf3 ({username='slave'} at 127.22.252.134:55213)
I20240501 14:09:38.929971 30297 tablet_copy_service.cc:161] P 1d0bb69611af4876a2b3b33c4766e13a: Beginning new tablet copy session on tablet 44949ddf1eac45cfa4fc8e1c532cf7bb from peer 5f8a0cfdfdca4148a75686dc88059cf3 at {username='slave'} at 127.22.252.134:55213: session id = 5f8a0cfdfdca4148a75686dc88059cf3-44949ddf1eac45cfa4fc8e1c532cf7bb
I20240501 14:09:38.931026 30297 tablet_copy_source_session.cc:205] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:38.933410 2691 tablet_bootstrap.cc:492] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:38.933502 2691 ts_tablet_manager.cc:1367] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:09:38.934247 30297 tablet_copy_service.cc:140] P 1d0bb69611af4876a2b3b33c4766e13a: Received BeginTabletCopySession request for tablet 0517ff3519644b489b912f8e5a36274c from peer 5f8a0cfdfdca4148a75686dc88059cf3 ({username='slave'} at 127.22.252.134:55213)
I20240501 14:09:38.934329 30297 tablet_copy_service.cc:161] P 1d0bb69611af4876a2b3b33c4766e13a: Beginning new tablet copy session on tablet 0517ff3519644b489b912f8e5a36274c from peer 5f8a0cfdfdca4148a75686dc88059cf3 at {username='slave'} at 127.22.252.134:55213: session id = 5f8a0cfdfdca4148a75686dc88059cf3-0517ff3519644b489b912f8e5a36274c
I20240501 14:09:38.934551 2691 raft_consensus.cc:348] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:38.934660 2691 raft_consensus.cc:374] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.934691 2691 raft_consensus.cc:729] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.935045 30297 tablet_copy_source_session.cc:205] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:38.934801 2691 consensus_queue.cc:260] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:38.935526 2691 ts_tablet_manager.cc:1398] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:38.935976 2696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44949ddf1eac45cfa4fc8e1c532cf7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.938786 2696 tablet_copy_client.cc:783] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:38.938973 2696 tablet_copy_client.cc:647] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:38.939329 2695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0517ff3519644b489b912f8e5a36274c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.940214 2421 raft_consensus.cc:482] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:38.940341 2421 raft_consensus.cc:504] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:38.940766 2421 leader_election.cc:288] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:38.941193 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65bc0e6ad1fa491e8e5ae225f86561f9" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:38.941306 32554 raft_consensus.cc:2454] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:38.941426 2696 tablet_copy_client.cc:523] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:38.941581 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65bc0e6ad1fa491e8e5ae225f86561f9" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:38.941655 2192 ts_tablet_manager.cc:1398] T b0ef35582c4f4d5c95199e00ccac92c9 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.013s user 0.000s sys 0.001s
I20240501 14:09:38.941710 348 raft_consensus.cc:2454] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:38.942420 32621 leader_election.cc:302] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:38.942716 2421 raft_consensus.cc:2781] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:38.942767 2421 raft_consensus.cc:482] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:38.942806 2421 raft_consensus.cc:3037] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.943081 2695 tablet_copy_client.cc:783] T 0517ff3519644b489b912f8e5a36274c P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:38.943135 2696 tablet_bootstrap.cc:492] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:38.943285 2695 tablet_copy_client.cc:647] T 0517ff3519644b489b912f8e5a36274c P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:38.943809 2421 raft_consensus.cc:504] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:38.944157 2421 leader_election.cc:288] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:38.944396 2695 tablet_copy_client.cc:523] T 0517ff3519644b489b912f8e5a36274c P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:38.945545 2695 tablet_bootstrap.cc:492] T 0517ff3519644b489b912f8e5a36274c P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:38.945878 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65bc0e6ad1fa491e8e5ae225f86561f9" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:38.945993 348 raft_consensus.cc:3037] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.946604 2420 raft_consensus.cc:482] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:38.946698 2420 raft_consensus.cc:504] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.946943 348 raft_consensus.cc:2454] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:38.947192 2420 leader_election.cc:288] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:38.947459 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65bc0e6ad1fa491e8e5ae225f86561f9" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:38.947549 32554 raft_consensus.cc:3037] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.947537 32620 leader_election.cc:302] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:38.947736 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ef35582c4f4d5c95199e00ccac92c9" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:38.947862 32415 raft_consensus.cc:2454] T b0ef35582c4f4d5c95199e00ccac92c9 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:38.947834 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ef35582c4f4d5c95199e00ccac92c9" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:38.947952 32553 raft_consensus.cc:2454] T b0ef35582c4f4d5c95199e00ccac92c9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:38.948200 2421 raft_consensus.cc:2781] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:38.948181 30343 leader_election.cc:302] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:38.948292 2421 raft_consensus.cc:686] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Becoming Leader. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
I20240501 14:09:38.948351 32554 raft_consensus.cc:2454] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:38.948444 2421 consensus_queue.cc:237] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:38.948772 2420 raft_consensus.cc:2781] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:38.948832 2420 raft_consensus.cc:482] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:38.948871 2420 raft_consensus.cc:3037] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.950001 2420 raft_consensus.cc:504] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.950341 2420 leader_election.cc:288] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:38.952383 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ef35582c4f4d5c95199e00ccac92c9" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:38.952474 32554 raft_consensus.cc:3037] T b0ef35582c4f4d5c95199e00ccac92c9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.953227 32554 raft_consensus.cc:2454] T b0ef35582c4f4d5c95199e00ccac92c9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:38.953666 30344 leader_election.cc:302] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:38.953698 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ef35582c4f4d5c95199e00ccac92c9" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:38.953792 32415 raft_consensus.cc:3037] T b0ef35582c4f4d5c95199e00ccac92c9 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.954360 29864 catalog_manager.cc:5484] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db935501e9104de5b15c8780681d6c38 (127.22.252.135). New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:38.954560 2678 tablet_bootstrap.cc:492] T c5d406c64b2e432a88e8c675b91dd144 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:38.954773 2678 tablet_bootstrap.cc:492] T c5d406c64b2e432a88e8c675b91dd144 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap complete.
I20240501 14:09:38.954783 32415 raft_consensus.cc:2454] T b0ef35582c4f4d5c95199e00ccac92c9 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:38.954878 2678 ts_tablet_manager.cc:1367] T c5d406c64b2e432a88e8c675b91dd144 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.039s user 0.024s sys 0.000s
I20240501 14:09:38.955636 2678 raft_consensus.cc:348] T c5d406c64b2e432a88e8c675b91dd144 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.955747 2678 raft_consensus.cc:729] T c5d406c64b2e432a88e8c675b91dd144 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.956341 2420 raft_consensus.cc:2781] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:38.956506 2420 raft_consensus.cc:686] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:38.956619 2420 consensus_queue.cc:237] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.957150 2678 consensus_queue.cc:260] T c5d406c64b2e432a88e8c675b91dd144 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.960127 29864 catalog_manager.cc:5484] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:38.960762 2678 ts_tablet_manager.cc:1398] T c5d406c64b2e432a88e8c675b91dd144 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 14:09:38.962105 30297 tablet_copy_service.cc:342] P 1d0bb69611af4876a2b3b33c4766e13a: Request end of tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-c5d406c64b2e432a88e8c675b91dd144 received from {username='slave'} at 127.22.252.134:55213
I20240501 14:09:38.962180 30297 tablet_copy_service.cc:433] P 1d0bb69611af4876a2b3b33c4766e13a: ending tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-c5d406c64b2e432a88e8c675b91dd144 on tablet c5d406c64b2e432a88e8c675b91dd144 with peer 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:09:38.977357 2695 tablet_bootstrap.cc:492] T 0517ff3519644b489b912f8e5a36274c P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:38.977623 2695 tablet_bootstrap.cc:492] T 0517ff3519644b489b912f8e5a36274c P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap complete.
I20240501 14:09:38.977771 2695 ts_tablet_manager.cc:1367] T 0517ff3519644b489b912f8e5a36274c P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.032s user 0.021s sys 0.004s
I20240501 14:09:38.978127 2695 raft_consensus.cc:348] T 0517ff3519644b489b912f8e5a36274c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.978246 2695 raft_consensus.cc:729] T 0517ff3519644b489b912f8e5a36274c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.978402 2695 consensus_queue.cc:260] T 0517ff3519644b489b912f8e5a36274c P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.979332 2695 ts_tablet_manager.cc:1398] T 0517ff3519644b489b912f8e5a36274c P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:38.979699 30297 tablet_copy_service.cc:342] P 1d0bb69611af4876a2b3b33c4766e13a: Request end of tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-0517ff3519644b489b912f8e5a36274c received from {username='slave'} at 127.22.252.134:55213
I20240501 14:09:38.979764 30297 tablet_copy_service.cc:433] P 1d0bb69611af4876a2b3b33c4766e13a: ending tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-0517ff3519644b489b912f8e5a36274c on tablet 0517ff3519644b489b912f8e5a36274c with peer 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:09:38.982889 2728 ts_tablet_manager.cc:902] T 66471362a1d24a8dab53f9cad512a73a P d1c217469ae4478ebd49e4ed9f7352d7: Initiating tablet copy from peer 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:38.983328 2723 ts_tablet_manager.cc:902] T f2541122e05f45c2843d06c5342c8d01 P db935501e9104de5b15c8780681d6c38: Initiating tablet copy from peer 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:38.983868 2723 tablet_copy_client.cc:308] T f2541122e05f45c2843d06c5342c8d01 P db935501e9104de5b15c8780681d6c38: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.136:45937
I20240501 14:09:38.983619 2728 tablet_copy_client.cc:308] T 66471362a1d24a8dab53f9cad512a73a P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.136:45937
I20240501 14:09:38.984546 369 tablet_copy_service.cc:140] P 9ecd907cf5634ac98f3f54f3dbec6849: Received BeginTabletCopySession request for tablet f2541122e05f45c2843d06c5342c8d01 from peer db935501e9104de5b15c8780681d6c38 ({username='slave'} at 127.22.252.135:37379)
I20240501 14:09:38.984679 369 tablet_copy_service.cc:161] P 9ecd907cf5634ac98f3f54f3dbec6849: Beginning new tablet copy session on tablet f2541122e05f45c2843d06c5342c8d01 from peer db935501e9104de5b15c8780681d6c38 at {username='slave'} at 127.22.252.135:37379: session id = db935501e9104de5b15c8780681d6c38-f2541122e05f45c2843d06c5342c8d01
I20240501 14:09:38.985126 2696 tablet_bootstrap.cc:492] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:38.985368 2696 tablet_bootstrap.cc:492] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap complete.
I20240501 14:09:38.985498 2696 ts_tablet_manager.cc:1367] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.042s user 0.020s sys 0.004s
I20240501 14:09:38.985813 2696 raft_consensus.cc:348] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.985898 2696 raft_consensus.cc:729] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.986023 369 tablet_copy_source_session.cc:205] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:38.986032 2696 consensus_queue.cc:260] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:38.986743 2696 ts_tablet_manager.cc:1398] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:38.987094 30297 tablet_copy_service.cc:342] P 1d0bb69611af4876a2b3b33c4766e13a: Request end of tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-44949ddf1eac45cfa4fc8e1c532cf7bb received from {username='slave'} at 127.22.252.134:55213
I20240501 14:09:38.987161 30297 tablet_copy_service.cc:433] P 1d0bb69611af4876a2b3b33c4766e13a: ending tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-44949ddf1eac45cfa4fc8e1c532cf7bb on tablet 44949ddf1eac45cfa4fc8e1c532cf7bb with peer 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:09:38.987536 369 tablet_copy_service.cc:140] P 9ecd907cf5634ac98f3f54f3dbec6849: Received BeginTabletCopySession request for tablet 66471362a1d24a8dab53f9cad512a73a from peer d1c217469ae4478ebd49e4ed9f7352d7 ({username='slave'} at 127.22.252.132:39439)
I20240501 14:09:38.987614 369 tablet_copy_service.cc:161] P 9ecd907cf5634ac98f3f54f3dbec6849: Beginning new tablet copy session on tablet 66471362a1d24a8dab53f9cad512a73a from peer d1c217469ae4478ebd49e4ed9f7352d7 at {username='slave'} at 127.22.252.132:39439: session id = d1c217469ae4478ebd49e4ed9f7352d7-66471362a1d24a8dab53f9cad512a73a
I20240501 14:09:38.987699 2723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2541122e05f45c2843d06c5342c8d01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.988389 369 tablet_copy_source_session.cc:205] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:38.989995 2723 tablet_copy_client.cc:783] T f2541122e05f45c2843d06c5342c8d01 P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:38.990053 2666 raft_consensus.cc:482] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:38.990128 2666 raft_consensus.cc:504] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:38.990190 2723 tablet_copy_client.cc:647] T f2541122e05f45c2843d06c5342c8d01 P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:38.990432 2666 leader_election.cc:288] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:38.990855 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb8f294e9704c549e1e0590b1429fe5" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:38.990978 30146 raft_consensus.cc:2454] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:38.991379 32490 leader_election.cc:302] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:38.991847 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb8f294e9704c549e1e0590b1429fe5" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:38.991961 348 raft_consensus.cc:2454] T 7cb8f294e9704c549e1e0590b1429fe5 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:38.992070 2723 tablet_copy_client.cc:523] T f2541122e05f45c2843d06c5342c8d01 P db935501e9104de5b15c8780681d6c38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:38.992350 2666 raft_consensus.cc:2781] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:38.992404 2666 raft_consensus.cc:482] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:38.992436 2666 raft_consensus.cc:3037] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.993340 2666 raft_consensus.cc:504] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:38.993507 2723 tablet_bootstrap.cc:492] T f2541122e05f45c2843d06c5342c8d01 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:38.993549 2728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66471362a1d24a8dab53f9cad512a73a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.993636 2666 leader_election.cc:288] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:38.994422 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb8f294e9704c549e1e0590b1429fe5" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:38.994534 30145 raft_consensus.cc:3037] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.994546 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb8f294e9704c549e1e0590b1429fe5" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:38.994633 348 raft_consensus.cc:3037] T 7cb8f294e9704c549e1e0590b1429fe5 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.995450 30145 raft_consensus.cc:2454] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:38.995501 348 raft_consensus.cc:2454] T 7cb8f294e9704c549e1e0590b1429fe5 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:38.995721 29866 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:38.996858 2728 tablet_copy_client.cc:783] T 66471362a1d24a8dab53f9cad512a73a P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:38.997059 2728 tablet_copy_client.cc:647] T 66471362a1d24a8dab53f9cad512a73a P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:38.997875 32490 leader_election.cc:302] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:39.004757 2350 tablet.cc:1722] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:39.006034 2351 tablet.cc:1722] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:39.008708 2643 tablet.cc:1722] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:39.009778 2723 tablet_bootstrap.cc:492] T f2541122e05f45c2843d06c5342c8d01 P db935501e9104de5b15c8780681d6c38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:39.009996 29864 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=30dffeac8b584dedb2b82736abfdc127] alter complete (version 3)
I20240501 14:09:39.010040 2723 tablet_bootstrap.cc:492] T f2541122e05f45c2843d06c5342c8d01 P db935501e9104de5b15c8780681d6c38: Bootstrap complete.
I20240501 14:09:39.010198 2723 ts_tablet_manager.cc:1367] T f2541122e05f45c2843d06c5342c8d01 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.004s
I20240501 14:09:39.011379 2666 raft_consensus.cc:2781] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:39.011467 2666 raft_consensus.cc:686] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:09:39.011583 2666 consensus_queue.cc:237] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:39.012504 2723 raft_consensus.cc:348] T f2541122e05f45c2843d06c5342c8d01 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:39.012609 2723 raft_consensus.cc:729] T f2541122e05f45c2843d06c5342c8d01 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.012960 2723 consensus_queue.cc:260] T f2541122e05f45c2843d06c5342c8d01 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:39.013409 2723 ts_tablet_manager.cc:1398] T f2541122e05f45c2843d06c5342c8d01 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:39.013350 29864 catalog_manager.cc:5484] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:39.014061 369 tablet_copy_service.cc:342] P 9ecd907cf5634ac98f3f54f3dbec6849: Request end of tablet copy session db935501e9104de5b15c8780681d6c38-f2541122e05f45c2843d06c5342c8d01 received from {username='slave'} at 127.22.252.135:37379
I20240501 14:09:39.014145 369 tablet_copy_service.cc:433] P 9ecd907cf5634ac98f3f54f3dbec6849: ending tablet copy session db935501e9104de5b15c8780681d6c38-f2541122e05f45c2843d06c5342c8d01 on tablet f2541122e05f45c2843d06c5342c8d01 with peer db935501e9104de5b15c8780681d6c38
I20240501 14:09:39.015185 2728 tablet_copy_client.cc:523] T 66471362a1d24a8dab53f9cad512a73a P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:39.017026 2745 ts_tablet_manager.cc:902] T 35dd2c9d436448c482ef0b3fac3a3b64 P d1c217469ae4478ebd49e4ed9f7352d7: Initiating tablet copy from peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:39.017539 2745 tablet_copy_client.cc:308] T 35dd2c9d436448c482ef0b3fac3a3b64 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.133:46265
I20240501 14:09:39.018126 32443 tablet_copy_service.cc:140] P a91029b90bf9415a89a2f70736759ad3: Received BeginTabletCopySession request for tablet 35dd2c9d436448c482ef0b3fac3a3b64 from peer d1c217469ae4478ebd49e4ed9f7352d7 ({username='slave'} at 127.22.252.132:53277)
I20240501 14:09:39.018257 32443 tablet_copy_service.cc:161] P a91029b90bf9415a89a2f70736759ad3: Beginning new tablet copy session on tablet 35dd2c9d436448c482ef0b3fac3a3b64 from peer d1c217469ae4478ebd49e4ed9f7352d7 at {username='slave'} at 127.22.252.132:53277: session id = d1c217469ae4478ebd49e4ed9f7352d7-35dd2c9d436448c482ef0b3fac3a3b64
I20240501 14:09:39.018550 2728 tablet_bootstrap.cc:492] T 66471362a1d24a8dab53f9cad512a73a P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:39.019567 32443 tablet_copy_source_session.cc:205] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:39.020972 2745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35dd2c9d436448c482ef0b3fac3a3b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.023100 2745 tablet_copy_client.cc:783] T 35dd2c9d436448c482ef0b3fac3a3b64 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:39.023224 2745 tablet_copy_client.cc:647] T 35dd2c9d436448c482ef0b3fac3a3b64 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:39.025252 2745 tablet_copy_client.cc:523] T 35dd2c9d436448c482ef0b3fac3a3b64 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:39.026377 2745 tablet_bootstrap.cc:492] T 35dd2c9d436448c482ef0b3fac3a3b64 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:39.029820 2745 tablet_bootstrap.cc:492] T 35dd2c9d436448c482ef0b3fac3a3b64 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:39.030184 2745 tablet_bootstrap.cc:492] T 35dd2c9d436448c482ef0b3fac3a3b64 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap complete.
I20240501 14:09:39.030383 2745 ts_tablet_manager.cc:1367] T 35dd2c9d436448c482ef0b3fac3a3b64 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.005s
I20240501 14:09:39.030897 2745 raft_consensus.cc:348] T 35dd2c9d436448c482ef0b3fac3a3b64 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:39.031067 2745 raft_consensus.cc:729] T 35dd2c9d436448c482ef0b3fac3a3b64 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.031322 2745 consensus_queue.cc:260] T 35dd2c9d436448c482ef0b3fac3a3b64 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:39.031713 2745 ts_tablet_manager.cc:1398] T 35dd2c9d436448c482ef0b3fac3a3b64 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:39.033054 32443 tablet_copy_service.cc:342] P a91029b90bf9415a89a2f70736759ad3: Request end of tablet copy session d1c217469ae4478ebd49e4ed9f7352d7-35dd2c9d436448c482ef0b3fac3a3b64 received from {username='slave'} at 127.22.252.132:53277
I20240501 14:09:39.033149 32443 tablet_copy_service.cc:433] P a91029b90bf9415a89a2f70736759ad3: ending tablet copy session d1c217469ae4478ebd49e4ed9f7352d7-35dd2c9d436448c482ef0b3fac3a3b64 on tablet 35dd2c9d436448c482ef0b3fac3a3b64 with peer d1c217469ae4478ebd49e4ed9f7352d7
I20240501 14:09:39.034654 2728 tablet_bootstrap.cc:492] T 66471362a1d24a8dab53f9cad512a73a P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:39.034901 2728 tablet_bootstrap.cc:492] T 66471362a1d24a8dab53f9cad512a73a P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap complete.
I20240501 14:09:39.035069 2728 ts_tablet_manager.cc:1367] T 66471362a1d24a8dab53f9cad512a73a P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240501 14:09:39.035465 2728 raft_consensus.cc:348] T 66471362a1d24a8dab53f9cad512a73a P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:39.035575 2728 raft_consensus.cc:729] T 66471362a1d24a8dab53f9cad512a73a P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.035694 2728 consensus_queue.cc:260] T 66471362a1d24a8dab53f9cad512a73a P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:39.035931 2728 ts_tablet_manager.cc:1398] T 66471362a1d24a8dab53f9cad512a73a P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.036571 369 tablet_copy_service.cc:342] P 9ecd907cf5634ac98f3f54f3dbec6849: Request end of tablet copy session d1c217469ae4478ebd49e4ed9f7352d7-66471362a1d24a8dab53f9cad512a73a received from {username='slave'} at 127.22.252.132:39439
I20240501 14:09:39.036661 369 tablet_copy_service.cc:433] P 9ecd907cf5634ac98f3f54f3dbec6849: ending tablet copy session d1c217469ae4478ebd49e4ed9f7352d7-66471362a1d24a8dab53f9cad512a73a on tablet 66471362a1d24a8dab53f9cad512a73a with peer d1c217469ae4478ebd49e4ed9f7352d7
I20240501 14:09:39.049820 2723 ts_tablet_manager.cc:902] T 75b67ccb9a334b25ae71b553a2a84a02 P db935501e9104de5b15c8780681d6c38: Initiating tablet copy from peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:39.050308 2723 tablet_copy_client.cc:308] T 75b67ccb9a334b25ae71b553a2a84a02 P db935501e9104de5b15c8780681d6c38: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.131:43103
I20240501 14:09:39.050820 30297 tablet_copy_service.cc:140] P 1d0bb69611af4876a2b3b33c4766e13a: Received BeginTabletCopySession request for tablet 75b67ccb9a334b25ae71b553a2a84a02 from peer db935501e9104de5b15c8780681d6c38 ({username='slave'} at 127.22.252.135:53463)
I20240501 14:09:39.050935 30297 tablet_copy_service.cc:161] P 1d0bb69611af4876a2b3b33c4766e13a: Beginning new tablet copy session on tablet 75b67ccb9a334b25ae71b553a2a84a02 from peer db935501e9104de5b15c8780681d6c38 at {username='slave'} at 127.22.252.135:53463: session id = db935501e9104de5b15c8780681d6c38-75b67ccb9a334b25ae71b553a2a84a02
I20240501 14:09:39.051986 30297 tablet_copy_source_session.cc:205] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:39.053525 2723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b67ccb9a334b25ae71b553a2a84a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.056212 2723 tablet_copy_client.cc:783] T 75b67ccb9a334b25ae71b553a2a84a02 P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:39.056457 2723 tablet_copy_client.cc:647] T 75b67ccb9a334b25ae71b553a2a84a02 P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:39.057541 2723 tablet_copy_client.cc:523] T 75b67ccb9a334b25ae71b553a2a84a02 P db935501e9104de5b15c8780681d6c38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:39.058789 2723 tablet_bootstrap.cc:492] T 75b67ccb9a334b25ae71b553a2a84a02 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:39.080819 30256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:39.081836 329 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:39.082309 32534 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:39.087517 30387 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:39.101110 2723 tablet_bootstrap.cc:492] T 75b67ccb9a334b25ae71b553a2a84a02 P db935501e9104de5b15c8780681d6c38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:39.101549 2723 tablet_bootstrap.cc:492] T 75b67ccb9a334b25ae71b553a2a84a02 P db935501e9104de5b15c8780681d6c38: Bootstrap complete.
I20240501 14:09:39.101738 2723 ts_tablet_manager.cc:1367] T 75b67ccb9a334b25ae71b553a2a84a02 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.043s user 0.027s sys 0.001s
I20240501 14:09:39.101972 30125 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:39.102227 2723 raft_consensus.cc:348] T 75b67ccb9a334b25ae71b553a2a84a02 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:39.102387 2723 raft_consensus.cc:729] T 75b67ccb9a334b25ae71b553a2a84a02 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.102562 2723 consensus_queue.cc:260] T 75b67ccb9a334b25ae71b553a2a84a02 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:39.102839 2723 ts_tablet_manager.cc:1398] T 75b67ccb9a334b25ae71b553a2a84a02 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.103549 29993 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:39.103722 30297 tablet_copy_service.cc:342] P 1d0bb69611af4876a2b3b33c4766e13a: Request end of tablet copy session db935501e9104de5b15c8780681d6c38-75b67ccb9a334b25ae71b553a2a84a02 received from {username='slave'} at 127.22.252.135:53463
I20240501 14:09:39.103796 30297 tablet_copy_service.cc:433] P 1d0bb69611af4876a2b3b33c4766e13a: ending tablet copy session db935501e9104de5b15c8780681d6c38-75b67ccb9a334b25ae71b553a2a84a02 on tablet 75b67ccb9a334b25ae71b553a2a84a02 with peer db935501e9104de5b15c8780681d6c38
I20240501 14:09:39.106771 32400 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:39.114986 29866 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:39.115131 29866 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:39.117323 29866 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 02ab742a1e42416da3203b289ae3ba36
I20240501 14:09:39.117918 32400 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ab742a1e42416da3203b289ae3ba36 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:39 UTC) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:39.118028 32534 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ab742a1e42416da3203b289ae3ba36 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:39 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:39.118512 329 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ab742a1e42416da3203b289ae3ba36 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:39 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:39.120415 2789 tablet_replica.cc:330] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:39.120604 2789 raft_consensus.cc:2229] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:39.120746 2789 raft_consensus.cc:2258] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:39.121448 2789 ts_tablet_manager.cc:1875] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:39.124377 2789 ts_tablet_manager.cc:1888] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:39.124446 2789 log.cc:1196] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/02ab742a1e42416da3203b289ae3ba36
I20240501 14:09:39.124739 2789 ts_tablet_manager.cc:1909] T 02ab742a1e42416da3203b289ae3ba36 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:39.125380 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet 02ab742a1e42416da3203b289ae3ba36 (table rebalancer_dynamic_table_new_name [id=30dffeac8b584dedb2b82736abfdc127]) successfully deleted
I20240501 14:09:39.121778 2790 tablet_replica.cc:330] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:39.128500 2790 raft_consensus.cc:2229] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:39.128723 2790 raft_consensus.cc:2258] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:39.129443 2790 ts_tablet_manager.cc:1875] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:39.131677 2790 ts_tablet_manager.cc:1888] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:39.131774 2790 log.cc:1196] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/02ab742a1e42416da3203b289ae3ba36
I20240501 14:09:39.132076 2790 ts_tablet_manager.cc:1909] T 02ab742a1e42416da3203b289ae3ba36 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:09:39.132755 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 02ab742a1e42416da3203b289ae3ba36 (table rebalancer_dynamic_table_new_name [id=30dffeac8b584dedb2b82736abfdc127]) successfully deleted
I20240501 14:09:39.140990 2788 tablet_replica.cc:330] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:39.141208 2788 raft_consensus.cc:2229] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:39.141367 2788 raft_consensus.cc:2258] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:39.143483 32664 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:39.150024 2788 ts_tablet_manager.cc:1875] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:39.152166 2788 ts_tablet_manager.cc:1888] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:39.156297 2788 log.cc:1196] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/02ab742a1e42416da3203b289ae3ba36
I20240501 14:09:39.156874 2788 ts_tablet_manager.cc:1909] T 02ab742a1e42416da3203b289ae3ba36 P a91029b90bf9415a89a2f70736759ad3: Deleting consensus metadata
I20240501 14:09:39.157676 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 02ab742a1e42416da3203b289ae3ba36 (table rebalancer_dynamic_table_new_name [id=30dffeac8b584dedb2b82736abfdc127]) successfully deleted
I20240501 14:09:39.220350 29865 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:39.226195 29993 tablet_service.cc:1467] Processing CreateTablet for tablet fc6fc57312104cf49c9e75e752861009 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5f2b80ec24984abd87c47dc087f23781]), partition=
I20240501 14:09:39.226238 32400 tablet_service.cc:1467] Processing CreateTablet for tablet fc6fc57312104cf49c9e75e752861009 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5f2b80ec24984abd87c47dc087f23781]), partition=
I20240501 14:09:39.226488 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc6fc57312104cf49c9e75e752861009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.226497 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc6fc57312104cf49c9e75e752861009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.228736 2571 tablet_bootstrap.cc:492] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:39.228742 329 tablet_service.cc:1467] Processing CreateTablet for tablet fc6fc57312104cf49c9e75e752861009 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5f2b80ec24984abd87c47dc087f23781]), partition=
I20240501 14:09:39.228977 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc6fc57312104cf49c9e75e752861009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.229538 2571 tablet_bootstrap.cc:654] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.230312 2831 tablet_bootstrap.cc:492] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:39.230600 2571 tablet_bootstrap.cc:492] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:39.230679 2571 ts_tablet_manager.cc:1367] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:39.231037 2571 raft_consensus.cc:348] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:39.231144 2571 raft_consensus.cc:374] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.231176 2190 tablet_bootstrap.cc:492] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:39.231186 2571 raft_consensus.cc:729] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.231227 2831 tablet_bootstrap.cc:654] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.231302 2571 consensus_queue.cc:260] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:39.231580 2571 ts_tablet_manager.cc:1398] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.232028 2190 tablet_bootstrap.cc:654] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.234215 2190 tablet_bootstrap.cc:492] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:39.234308 2190 ts_tablet_manager.cc:1367] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:39.234736 2190 raft_consensus.cc:348] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:39.234848 2190 raft_consensus.cc:374] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.234880 2190 raft_consensus.cc:729] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.235010 2190 consensus_queue.cc:260] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:39.235286 2190 ts_tablet_manager.cc:1398] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:39.238361 2831 tablet_bootstrap.cc:492] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:39.238456 2831 ts_tablet_manager.cc:1367] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:39.238919 2831 raft_consensus.cc:348] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:39.239032 2831 raft_consensus.cc:374] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.239065 2831 raft_consensus.cc:729] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.239181 2831 consensus_queue.cc:260] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:39.239462 2831 ts_tablet_manager.cc:1398] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.258908 30256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:39.261919 32534 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:39.272172 32664 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:39.274436 32400 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:39.275173 329 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:39.275440 29993 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:39.280908 30125 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:39.281567 30387 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:39.327827 32553 raft_consensus.cc:1203] T 0517ff3519644b489b912f8e5a36274c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:39.330083 32553 raft_consensus.cc:1203] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:39.345700 30408 consensus_queue.cc:237] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:39.346915 30277 raft_consensus.cc:1261] T 53c51f3845c94f47b043621d98f70229 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:39.346937 32415 raft_consensus.cc:1261] T 53c51f3845c94f47b043621d98f70229 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:39.347288 2720 consensus_queue.cc:1035] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:39.347487 2420 consensus_queue.cc:1035] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:39.349334 2864 raft_consensus.cc:2932] T 53c51f3845c94f47b043621d98f70229 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:39.349714 32415 raft_consensus.cc:2932] T 53c51f3845c94f47b043621d98f70229 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:39.349762 30277 raft_consensus.cc:2932] T 53c51f3845c94f47b043621d98f70229 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:39.352380 29866 catalog_manager.cc:5484] T 53c51f3845c94f47b043621d98f70229 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:39.355414 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 53c51f3845c94f47b043621d98f70229 with delete_type TABLET_DATA_TOMBSTONED (TS d35c16dcb506426993811bb2519d7db3 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:39.356024 2869 tablet_replica.cc:330] T 53c51f3845c94f47b043621d98f70229 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:39.356215 2869 raft_consensus.cc:2229] T 53c51f3845c94f47b043621d98f70229 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:39.356559 2869 raft_consensus.cc:2258] T 53c51f3845c94f47b043621d98f70229 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:39.357205 2869 ts_tablet_manager.cc:1875] T 53c51f3845c94f47b043621d98f70229 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:39.359486 2869 ts_tablet_manager.cc:1888] T 53c51f3845c94f47b043621d98f70229 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:39.359555 2869 log.cc:1196] T 53c51f3845c94f47b043621d98f70229 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/53c51f3845c94f47b043621d98f70229
I20240501 14:09:39.360018 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 53c51f3845c94f47b043621d98f70229 (table rebalance_test_table_3 [id=b7b3f5bc4b6649c5a97ec9f02da4e967]) successfully deleted
I20240501 14:09:39.370139 2008 consensus_queue.cc:1035] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.373764 2421 consensus_queue.cc:1035] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.378661 2361 raft_consensus.cc:482] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:39.378762 2361 raft_consensus.cc:504] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:39.379127 2361 leader_election.cc:288] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:39.379540 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6fc57312104cf49c9e75e752861009" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:39.379554 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6fc57312104cf49c9e75e752861009" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:39.379653 348 raft_consensus.cc:2454] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:39.379667 32415 raft_consensus.cc:2454] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:39.379912 29950 leader_election.cc:302] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:39.380064 2361 raft_consensus.cc:2781] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:39.380111 2361 raft_consensus.cc:482] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:39.380149 2361 raft_consensus.cc:3037] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.381244 2361 raft_consensus.cc:504] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:39.381610 2361 leader_election.cc:288] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:39.382347 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6fc57312104cf49c9e75e752861009" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:39.382347 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6fc57312104cf49c9e75e752861009" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:39.382447 348 raft_consensus.cc:3037] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.382447 32415 raft_consensus.cc:3037] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.383328 348 raft_consensus.cc:2454] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:39.383328 32415 raft_consensus.cc:2454] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:39.383713 29950 leader_election.cc:302] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:39.384227 2361 raft_consensus.cc:2781] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:39.384342 2361 raft_consensus.cc:686] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:39.384479 2361 consensus_queue.cc:237] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:39.386758 29873 catalog_manager.cc:5484] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:39.398463 2666 consensus_queue.cc:1035] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.403488 30408 raft_consensus.cc:1203] T 66471362a1d24a8dab53f9cad512a73a P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:39.405650 2913 consensus_queue.cc:1035] T 7cb8f294e9704c549e1e0590b1429fe5 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.422744 32553 raft_consensus.cc:1203] T c5d406c64b2e432a88e8c675b91dd144 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:39.445886 30256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:39.452250 30125 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:39.452925 32534 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:39.458633 2720 consensus_queue.cc:1035] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.460378 329 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:39.462546 2864 consensus_queue.cc:1035] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.464658 29993 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:39.465634 30406 raft_consensus.cc:1203] T 35dd2c9d436448c482ef0b3fac3a3b64 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:39.468896 32664 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:39.472380 30387 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:39.476084 32685 raft_consensus.cc:1203] T f2541122e05f45c2843d06c5342c8d01 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:39.489164 32400 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:39.523787 29872 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:39.528164 32415 raft_consensus.cc:1261] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:39.528184 348 raft_consensus.cc:1261] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:39.528721 2361 consensus_queue.cc:1035] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.528983 2361 consensus_queue.cc:1035] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.531908 29871 catalog_manager.cc:5576] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937) does not have the latest schema for tablet fc6fc57312104cf49c9e75e752861009 (table rebalancer_dynamic_table [id=5f2b80ec24984abd87c47dc087f23781]). Expected version 1 got 0
I20240501 14:09:39.532704 29869 catalog_manager.cc:5576] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265) does not have the latest schema for tablet fc6fc57312104cf49c9e75e752861009 (table rebalancer_dynamic_table [id=5f2b80ec24984abd87c47dc087f23781]). Expected version 1 got 0
I20240501 14:09:39.534426 2946 tablet.cc:1722] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:39.534616 2948 tablet.cc:1722] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:39.534920 2946 tablet.cc:1620] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:39.534925 2948 tablet.cc:1620] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:39.535676 29872 catalog_manager.cc:5576] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937) does not have the latest schema for tablet fc6fc57312104cf49c9e75e752861009 (table rebalancer_dynamic_table [id=5f2b80ec24984abd87c47dc087f23781]). Expected version 1 got 0
I20240501 14:09:39.536170 2941 mvcc.cc:205] Tried to move back new op lower bound from 7022889285743112192 to 7022889285160333312. Current Snapshot: MvccSnapshot[applied={T|T < 7022889285743112192}]
I20240501 14:09:39.539268 29871 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=5f2b80ec24984abd87c47dc087f23781] alter complete (version 1)
I20240501 14:09:39.539462 2945 tablet.cc:1722] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:39.539762 2945 tablet.cc:1620] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e: MemRowSet was empty: no flush needed.
I20240501 14:09:39.539886 29872 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:39.545619 30145 consensus_queue.cc:237] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:39.546059 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 290ae9a8a79345f88111981d96c60f67 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=02ac9598abf44cafb0f816aa3165fb2c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.546348 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 290ae9a8a79345f88111981d96c60f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.547190 329 tablet_service.cc:1467] Processing CreateTablet for tablet 290ae9a8a79345f88111981d96c60f67 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=02ac9598abf44cafb0f816aa3165fb2c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.547255 328 tablet_service.cc:1467] Processing CreateTablet for tablet 33de572fa35c40109cb70ee3287829f2 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=02ac9598abf44cafb0f816aa3165fb2c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.547452 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 290ae9a8a79345f88111981d96c60f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.547488 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 712044f41112481a8cc0a88afd0a916a (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=02ac9598abf44cafb0f816aa3165fb2c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.547717 32402 tablet_service.cc:1467] Processing CreateTablet for tablet 33de572fa35c40109cb70ee3287829f2 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=02ac9598abf44cafb0f816aa3165fb2c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.547720 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 712044f41112481a8cc0a88afd0a916a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.548717 2955 tablet_bootstrap.cc:492] T 290ae9a8a79345f88111981d96c60f67 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:39.549011 30125 tablet_service.cc:1467] Processing CreateTablet for tablet 290ae9a8a79345f88111981d96c60f67 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=02ac9598abf44cafb0f816aa3165fb2c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.549257 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 290ae9a8a79345f88111981d96c60f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.550029 30256 tablet_service.cc:1467] Processing CreateTablet for tablet 712044f41112481a8cc0a88afd0a916a (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=02ac9598abf44cafb0f816aa3165fb2c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.550222 30013 raft_consensus.cc:1261] T 18918995fb03421b9dfb4d05369c1ce8 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:39.550326 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 712044f41112481a8cc0a88afd0a916a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.550405 2955 tablet_bootstrap.cc:654] T 290ae9a8a79345f88111981d96c60f67 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.550482 2571 tablet_bootstrap.cc:492] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:39.551417 2571 tablet_bootstrap.cc:654] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.551472 2955 tablet_bootstrap.cc:492] T 290ae9a8a79345f88111981d96c60f67 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:39.551549 2955 ts_tablet_manager.cc:1367] T 290ae9a8a79345f88111981d96c60f67 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:39.551638 30124 tablet_service.cc:1467] Processing CreateTablet for tablet 33de572fa35c40109cb70ee3287829f2 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=02ac9598abf44cafb0f816aa3165fb2c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.551872 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33de572fa35c40109cb70ee3287829f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.551870 2190 tablet_bootstrap.cc:492] T 290ae9a8a79345f88111981d96c60f67 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:39.552008 2955 raft_consensus.cc:348] T 290ae9a8a79345f88111981d96c60f67 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:39.552102 2955 raft_consensus.cc:374] T 290ae9a8a79345f88111981d96c60f67 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.552132 2955 raft_consensus.cc:729] T 290ae9a8a79345f88111981d96c60f67 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.552736 2190 tablet_bootstrap.cc:654] T 290ae9a8a79345f88111981d96c60f67 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.552747 2955 consensus_queue.cc:260] T 290ae9a8a79345f88111981d96c60f67 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:39.552843 32402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33de572fa35c40109cb70ee3287829f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.553048 2955 ts_tablet_manager.cc:1398] T 290ae9a8a79345f88111981d96c60f67 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.553068 348 raft_consensus.cc:1261] T 18918995fb03421b9dfb4d05369c1ce8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:39.553329 2958 tablet_bootstrap.cc:492] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:39.554049 2404 consensus_queue.cc:1035] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:39.554407 2404 consensus_queue.cc:1035] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:39.554455 2958 tablet_bootstrap.cc:654] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.555858 2958 tablet_bootstrap.cc:492] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:39.555943 2958 ts_tablet_manager.cc:1367] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:39.556423 2958 raft_consensus.cc:348] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:39.556514 2958 raft_consensus.cc:374] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.556545 2958 raft_consensus.cc:729] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.556644 2958 consensus_queue.cc:260] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:39.556892 2958 ts_tablet_manager.cc:1398] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.557067 2958 tablet_bootstrap.cc:492] T 33de572fa35c40109cb70ee3287829f2 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:39.557334 328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33de572fa35c40109cb70ee3287829f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.557886 2958 tablet_bootstrap.cc:654] T 33de572fa35c40109cb70ee3287829f2 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.557982 2965 tablet_bootstrap.cc:492] T 712044f41112481a8cc0a88afd0a916a P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:39.559659 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 712044f41112481a8cc0a88afd0a916a (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=02ac9598abf44cafb0f816aa3165fb2c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.559808 2190 tablet_bootstrap.cc:492] T 290ae9a8a79345f88111981d96c60f67 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:39.559891 2190 ts_tablet_manager.cc:1367] T 290ae9a8a79345f88111981d96c60f67 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:09:39.559944 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 712044f41112481a8cc0a88afd0a916a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.560346 2958 tablet_bootstrap.cc:492] T 33de572fa35c40109cb70ee3287829f2 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:39.560411 2958 ts_tablet_manager.cc:1367] T 33de572fa35c40109cb70ee3287829f2 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:39.560355 2190 raft_consensus.cc:348] T 290ae9a8a79345f88111981d96c60f67 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:39.560456 2190 raft_consensus.cc:374] T 290ae9a8a79345f88111981d96c60f67 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.560487 2190 raft_consensus.cc:729] T 290ae9a8a79345f88111981d96c60f67 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.560597 2190 consensus_queue.cc:260] T 290ae9a8a79345f88111981d96c60f67 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:39.560775 2958 raft_consensus.cc:348] T 33de572fa35c40109cb70ee3287829f2 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:39.560849 2190 ts_tablet_manager.cc:1398] T 290ae9a8a79345f88111981d96c60f67 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:39.560863 2958 raft_consensus.cc:374] T 33de572fa35c40109cb70ee3287829f2 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.560895 2958 raft_consensus.cc:729] T 33de572fa35c40109cb70ee3287829f2 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.560997 2190 tablet_bootstrap.cc:492] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:39.560997 2958 consensus_queue.cc:260] T 33de572fa35c40109cb70ee3287829f2 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:39.561293 2958 ts_tablet_manager.cc:1398] T 33de572fa35c40109cb70ee3287829f2 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:39.561849 2190 tablet_bootstrap.cc:654] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.562857 2967 tablet_bootstrap.cc:492] T 712044f41112481a8cc0a88afd0a916a P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:39.563592 2960 raft_consensus.cc:2932] T 18918995fb03421b9dfb4d05369c1ce8 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:39.564651 348 raft_consensus.cc:2932] T 18918995fb03421b9dfb4d05369c1ce8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:39.564792 30013 raft_consensus.cc:2932] T 18918995fb03421b9dfb4d05369c1ce8 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:39.565155 2967 tablet_bootstrap.cc:654] T 712044f41112481a8cc0a88afd0a916a P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.566197 2967 tablet_bootstrap.cc:492] T 712044f41112481a8cc0a88afd0a916a P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:39.566272 2967 ts_tablet_manager.cc:1367] T 712044f41112481a8cc0a88afd0a916a P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:39.566701 2967 raft_consensus.cc:348] T 712044f41112481a8cc0a88afd0a916a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:39.566802 2967 raft_consensus.cc:374] T 712044f41112481a8cc0a88afd0a916a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.566844 2967 raft_consensus.cc:729] T 712044f41112481a8cc0a88afd0a916a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.566973 2967 consensus_queue.cc:260] T 712044f41112481a8cc0a88afd0a916a P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:39.567102 2965 tablet_bootstrap.cc:654] T 712044f41112481a8cc0a88afd0a916a P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.567279 2967 ts_tablet_manager.cc:1398] T 712044f41112481a8cc0a88afd0a916a P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.567706 2571 tablet_bootstrap.cc:492] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:39.567785 2571 ts_tablet_manager.cc:1367] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 14:09:39.568329 2571 raft_consensus.cc:348] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:39.568426 2571 raft_consensus.cc:374] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.568457 2571 raft_consensus.cc:729] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.568578 2571 consensus_queue.cc:260] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:39.568848 2571 ts_tablet_manager.cc:1398] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.569000 2571 tablet_bootstrap.cc:492] T 33de572fa35c40109cb70ee3287829f2 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:39.569662 2404 raft_consensus.cc:482] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:39.569741 2404 raft_consensus.cc:504] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:39.569860 2571 tablet_bootstrap.cc:654] T 33de572fa35c40109cb70ee3287829f2 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.570077 2404 leader_election.cc:288] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:39.570425 29869 catalog_manager.cc:5484] T 18918995fb03421b9dfb4d05369c1ce8 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:39.570807 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290ae9a8a79345f88111981d96c60f67" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:39.570916 32553 raft_consensus.cc:2454] T 290ae9a8a79345f88111981d96c60f67 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:39.571841 30082 leader_election.cc:302] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:39.572470 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290ae9a8a79345f88111981d96c60f67" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:39.572580 348 raft_consensus.cc:2454] T 290ae9a8a79345f88111981d96c60f67 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:39.573294 2404 raft_consensus.cc:2781] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:39.573347 2190 tablet_bootstrap.cc:492] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:39.573347 2404 raft_consensus.cc:482] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:39.573388 2404 raft_consensus.cc:3037] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.573413 2190 ts_tablet_manager.cc:1367] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240501 14:09:39.573906 2190 raft_consensus.cc:348] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:39.574019 2190 raft_consensus.cc:374] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.574050 2190 raft_consensus.cc:729] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.574167 2190 consensus_queue.cc:260] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:39.574251 2404 raft_consensus.cc:504] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:39.574430 2190 ts_tablet_manager.cc:1398] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.574555 2404 leader_election.cc:288] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:39.574699 30256 tablet_service.cc:1514] Processing DeleteTablet for tablet 18918995fb03421b9dfb4d05369c1ce8 with delete_type TABLET_DATA_TOMBSTONED (TS 1d0bb69611af4876a2b3b33c4766e13a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:39.574986 2973 tablet_replica.cc:330] T 18918995fb03421b9dfb4d05369c1ce8 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:39.575167 2973 raft_consensus.cc:2229] T 18918995fb03421b9dfb4d05369c1ce8 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:39.575248 2571 tablet_bootstrap.cc:492] T 33de572fa35c40109cb70ee3287829f2 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:39.575322 2571 ts_tablet_manager.cc:1367] T 33de572fa35c40109cb70ee3287829f2 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:39.575474 2973 raft_consensus.cc:2258] T 18918995fb03421b9dfb4d05369c1ce8 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:39.575631 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290ae9a8a79345f88111981d96c60f67" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:39.575726 32553 raft_consensus.cc:3037] T 290ae9a8a79345f88111981d96c60f67 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.575692 2571 raft_consensus.cc:348] T 33de572fa35c40109cb70ee3287829f2 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:39.575790 2571 raft_consensus.cc:374] T 33de572fa35c40109cb70ee3287829f2 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.575829 2571 raft_consensus.cc:729] T 33de572fa35c40109cb70ee3287829f2 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.575950 2571 consensus_queue.cc:260] T 33de572fa35c40109cb70ee3287829f2 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:39.576056 2973 ts_tablet_manager.cc:1875] T 18918995fb03421b9dfb4d05369c1ce8 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:39.576184 2571 ts_tablet_manager.cc:1398] T 33de572fa35c40109cb70ee3287829f2 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.576613 32553 raft_consensus.cc:2454] T 290ae9a8a79345f88111981d96c60f67 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:39.577057 30082 leader_election.cc:302] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:39.577282 2969 raft_consensus.cc:2781] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:39.577461 2969 raft_consensus.cc:686] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:39.577618 2969 consensus_queue.cc:237] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:39.578138 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290ae9a8a79345f88111981d96c60f67" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:39.578243 348 raft_consensus.cc:3037] T 290ae9a8a79345f88111981d96c60f67 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.579089 348 raft_consensus.cc:2454] T 290ae9a8a79345f88111981d96c60f67 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:39.568316 2965 tablet_bootstrap.cc:492] T 712044f41112481a8cc0a88afd0a916a P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:39.579667 2965 ts_tablet_manager.cc:1367] T 712044f41112481a8cc0a88afd0a916a P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240501 14:09:39.580042 29872 catalog_manager.cc:5484] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:39.580144 2965 raft_consensus.cc:348] T 712044f41112481a8cc0a88afd0a916a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:39.580289 2965 raft_consensus.cc:374] T 712044f41112481a8cc0a88afd0a916a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.580334 2965 raft_consensus.cc:729] T 712044f41112481a8cc0a88afd0a916a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.580451 2965 consensus_queue.cc:260] T 712044f41112481a8cc0a88afd0a916a P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:39.580790 2965 ts_tablet_manager.cc:1398] T 712044f41112481a8cc0a88afd0a916a P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.582168 2973 ts_tablet_manager.cc:1888] T 18918995fb03421b9dfb4d05369c1ce8 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:39.582273 2973 log.cc:1196] T 18918995fb03421b9dfb4d05369c1ce8 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/18918995fb03421b9dfb4d05369c1ce8
I20240501 14:09:39.582846 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet 18918995fb03421b9dfb4d05369c1ce8 (table rebalance_test_table_13 [id=422856ed458a4c52950e8e3e49019a55]) successfully deleted
I20240501 14:09:39.600211 32685 raft_consensus.cc:1203] T 75b67ccb9a334b25ae71b553a2a84a02 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:39.643981 29870 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:39.650131 2945 tablet.cc:1722] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:39.650482 2945 tablet.cc:1620] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e: MemRowSet was empty: no flush needed.
I20240501 14:09:39.651867 2946 tablet.cc:1722] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:39.652151 2946 tablet.cc:1620] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:39.652745 2948 tablet.cc:1722] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:39.653033 2948 tablet.cc:1620] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:39.660602 29866 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=5f2b80ec24984abd87c47dc087f23781] alter complete (version 2)
I20240501 14:09:39.704380 2203 raft_consensus.cc:482] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:39.704493 2203 raft_consensus.cc:504] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:39.704849 2203 leader_election.cc:288] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:39.705144 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33de572fa35c40109cb70ee3287829f2" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:39.705165 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33de572fa35c40109cb70ee3287829f2" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:39.705273 30145 raft_consensus.cc:2454] T 33de572fa35c40109cb70ee3287829f2 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:39.705289 32415 raft_consensus.cc:2454] T 33de572fa35c40109cb70ee3287829f2 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:39.705543 32753 leader_election.cc:302] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:39.705742 2203 raft_consensus.cc:2781] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:39.705801 2203 raft_consensus.cc:482] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:39.705838 2203 raft_consensus.cc:3037] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.706674 2203 raft_consensus.cc:504] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:39.707005 2203 leader_election.cc:288] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:39.707288 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33de572fa35c40109cb70ee3287829f2" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:39.707379 32415 raft_consensus.cc:3037] T 33de572fa35c40109cb70ee3287829f2 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.707399 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33de572fa35c40109cb70ee3287829f2" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:39.707510 30145 raft_consensus.cc:3037] T 33de572fa35c40109cb70ee3287829f2 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.708135 32415 raft_consensus.cc:2454] T 33de572fa35c40109cb70ee3287829f2 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:39.708339 30145 raft_consensus.cc:2454] T 33de572fa35c40109cb70ee3287829f2 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:39.708582 32752 leader_election.cc:302] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:39.708763 2203 raft_consensus.cc:2781] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:39.708859 2203 raft_consensus.cc:686] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:39.709023 2203 consensus_queue.cc:237] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:39.710702 29872 catalog_manager.cc:5484] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:39.763453 29872 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:39.768621 2945 tablet.cc:1722] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:39.769012 2948 tablet.cc:1722] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:39.769186 2946 tablet.cc:1722] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:39.773443 29865 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=5f2b80ec24984abd87c47dc087f23781] alter complete (version 3)
I20240501 14:09:39.876339 29872 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:39.876504 29872 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:39.878535 29872 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet fc6fc57312104cf49c9e75e752861009
I20240501 14:09:39.879120 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet fc6fc57312104cf49c9e75e752861009 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:39 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:39.879156 328 tablet_service.cc:1514] Processing DeleteTablet for tablet fc6fc57312104cf49c9e75e752861009 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:39 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:39.879451 3019 tablet_replica.cc:330] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:39.879532 32402 tablet_service.cc:1514] Processing DeleteTablet for tablet fc6fc57312104cf49c9e75e752861009 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:39 UTC) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:39.879612 3019 raft_consensus.cc:2229] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:39.879745 3019 raft_consensus.cc:2258] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:39.879792 3020 tablet_replica.cc:330] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:39.879907 3020 raft_consensus.cc:2229] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:39.879981 3018 tablet_replica.cc:330] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:39.880067 3020 raft_consensus.cc:2258] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:39.880125 3018 raft_consensus.cc:2229] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:39.880647 3019 ts_tablet_manager.cc:1875] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:39.880697 3020 ts_tablet_manager.cc:1875] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:39.882864 3019 ts_tablet_manager.cc:1888] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:39.882864 3020 ts_tablet_manager.cc:1888] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:39.882941 3020 log.cc:1196] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/fc6fc57312104cf49c9e75e752861009
I20240501 14:09:39.882941 3019 log.cc:1196] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/fc6fc57312104cf49c9e75e752861009
I20240501 14:09:39.883221 3019 ts_tablet_manager.cc:1909] T fc6fc57312104cf49c9e75e752861009 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:09:39.883229 3020 ts_tablet_manager.cc:1909] T fc6fc57312104cf49c9e75e752861009 P a91029b90bf9415a89a2f70736759ad3: Deleting consensus metadata
I20240501 14:09:39.883534 3018 raft_consensus.cc:2258] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:39.883914 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet fc6fc57312104cf49c9e75e752861009 (table rebalancer_dynamic_table_new_name [id=5f2b80ec24984abd87c47dc087f23781]) successfully deleted
I20240501 14:09:39.884073 3018 ts_tablet_manager.cc:1875] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:39.884081 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet fc6fc57312104cf49c9e75e752861009 (table rebalancer_dynamic_table_new_name [id=5f2b80ec24984abd87c47dc087f23781]) successfully deleted
I20240501 14:09:39.886157 3018 ts_tablet_manager.cc:1888] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:39.886217 3018 log.cc:1196] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/fc6fc57312104cf49c9e75e752861009
I20240501 14:09:39.886495 3018 ts_tablet_manager.cc:1909] T fc6fc57312104cf49c9e75e752861009 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:39.887063 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet fc6fc57312104cf49c9e75e752861009 (table rebalancer_dynamic_table_new_name [id=5f2b80ec24984abd87c47dc087f23781]) successfully deleted
I20240501 14:09:39.945966 2944 raft_consensus.cc:482] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:39.946079 2944 raft_consensus.cc:504] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:39.946578 2944 leader_election.cc:288] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:39.946885 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712044f41112481a8cc0a88afd0a916a" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:39.946929 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712044f41112481a8cc0a88afd0a916a" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:39.947008 30264 raft_consensus.cc:2454] T 712044f41112481a8cc0a88afd0a916a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:39.947059 32685 raft_consensus.cc:2454] T 712044f41112481a8cc0a88afd0a916a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:39.947289 32358 leader_election.cc:302] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:39.947435 2944 raft_consensus.cc:2781] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:39.947491 2944 raft_consensus.cc:482] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:39.947540 2944 raft_consensus.cc:3037] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.948524 2944 raft_consensus.cc:504] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:39.948834 2944 leader_election.cc:288] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:39.949137 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712044f41112481a8cc0a88afd0a916a" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:39.949254 32685 raft_consensus.cc:3037] T 712044f41112481a8cc0a88afd0a916a P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.949337 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712044f41112481a8cc0a88afd0a916a" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:39.949455 30264 raft_consensus.cc:3037] T 712044f41112481a8cc0a88afd0a916a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.950289 30264 raft_consensus.cc:2454] T 712044f41112481a8cc0a88afd0a916a P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:39.950316 32685 raft_consensus.cc:2454] T 712044f41112481a8cc0a88afd0a916a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:39.950614 32356 leader_election.cc:302] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:39.950807 2944 raft_consensus.cc:2781] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:39.950901 2944 raft_consensus.cc:686] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:39.951045 2944 consensus_queue.cc:237] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:39.952677 29872 catalog_manager.cc:5484] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:39.971592 30256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:39.981384 29993 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:39.981986 29872 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:39.983553 30387 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:39.984705 32534 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:39.986131 32402 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:39.998179 328 tablet_service.cc:1467] Processing CreateTablet for tablet 66ca98d099e24f0f86d4eca60011e107 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ab5ec2915fc044ad9585146a2afdd70e]), partition=
I20240501 14:09:39.998490 328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ca98d099e24f0f86d4eca60011e107. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.999512 32665 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:40.000293 30256 tablet_service.cc:1467] Processing CreateTablet for tablet 66ca98d099e24f0f86d4eca60011e107 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ab5ec2915fc044ad9585146a2afdd70e]), partition=
I20240501 14:09:40.000528 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ca98d099e24f0f86d4eca60011e107. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.000900 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 66ca98d099e24f0f86d4eca60011e107 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ab5ec2915fc044ad9585146a2afdd70e]), partition=
I20240501 14:09:40.001112 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ca98d099e24f0f86d4eca60011e107. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.003088 2190 tablet_bootstrap.cc:492] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:40.003901 2190 tablet_bootstrap.cc:654] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.004364 2965 tablet_bootstrap.cc:492] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:40.004557 2967 tablet_bootstrap.cc:492] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:40.005012 2190 tablet_bootstrap.cc:492] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:40.005103 2190 ts_tablet_manager.cc:1367] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:40.005177 2965 tablet_bootstrap.cc:654] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.005360 2967 tablet_bootstrap.cc:654] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.005596 2190 raft_consensus.cc:348] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:40.005700 2190 raft_consensus.cc:374] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.005731 2190 raft_consensus.cc:729] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.005870 2190 consensus_queue.cc:260] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:40.006145 2190 ts_tablet_manager.cc:1398] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:40.012105 328 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:40.035442 30124 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:40.042722 2967 tablet_bootstrap.cc:492] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:40.042824 2967 ts_tablet_manager.cc:1367] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.000s
I20240501 14:09:40.043263 2967 raft_consensus.cc:348] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:40.043368 2967 raft_consensus.cc:374] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.043394 2967 raft_consensus.cc:729] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.043504 2967 consensus_queue.cc:260] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:40.043754 2967 ts_tablet_manager.cc:1398] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.044296 2965 tablet_bootstrap.cc:492] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:40.044400 2965 ts_tablet_manager.cc:1367] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.000s
I20240501 14:09:40.044819 2965 raft_consensus.cc:348] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:40.044914 2965 raft_consensus.cc:374] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.044941 2965 raft_consensus.cc:729] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.045073 2965 consensus_queue.cc:260] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:40.045382 2965 ts_tablet_manager.cc:1398] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:40.045758 2421 raft_consensus.cc:482] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:40.045838 2421 raft_consensus.cc:504] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:40.046195 2421 leader_election.cc:288] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:40.046903 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ca98d099e24f0f86d4eca60011e107" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:40.047030 348 raft_consensus.cc:2454] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:40.047467 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ca98d099e24f0f86d4eca60011e107" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:40.047574 30264 raft_consensus.cc:2454] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:40.047840 32620 leader_election.cc:302] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:40.048130 2421 raft_consensus.cc:2781] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:40.048192 2421 raft_consensus.cc:482] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:40.048245 2421 raft_consensus.cc:3037] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.049233 2421 raft_consensus.cc:504] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:40.049537 2421 leader_election.cc:288] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:40.051124 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ca98d099e24f0f86d4eca60011e107" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:40.051265 30264 raft_consensus.cc:3037] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.051216 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ca98d099e24f0f86d4eca60011e107" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:40.051367 348 raft_consensus.cc:3037] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.052209 30264 raft_consensus.cc:2454] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:40.052322 348 raft_consensus.cc:2454] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:40.052780 32620 leader_election.cc:302] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:40.053011 2421 raft_consensus.cc:2781] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:40.053143 2421 raft_consensus.cc:686] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Becoming Leader. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
I20240501 14:09:40.053274 2421 consensus_queue.cc:237] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:40.055132 29865 catalog_manager.cc:5484] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db935501e9104de5b15c8780681d6c38 (127.22.252.135). New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:40.081224 30264 consensus_queue.cc:237] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:40.082441 32685 raft_consensus.cc:1261] T 75b67ccb9a334b25ae71b553a2a84a02 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:40.082515 30406 raft_consensus.cc:1261] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:40.082823 2956 consensus_queue.cc:1035] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:40.082955 2445 consensus_queue.cc:1035] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:40.084827 3062 raft_consensus.cc:2932] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:40.085172 32685 raft_consensus.cc:2932] T 75b67ccb9a334b25ae71b553a2a84a02 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:40.085206 30406 raft_consensus.cc:2932] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:40.088397 29865 catalog_manager.cc:5484] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: config changed from index 4 to 5, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:40.091203 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet 75b67ccb9a334b25ae71b553a2a84a02 with delete_type TABLET_DATA_TOMBSTONED (TS 93f2518fbadf4f9991cda93afacb363e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:40.091447 3018 tablet_replica.cc:330] T 75b67ccb9a334b25ae71b553a2a84a02 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:40.091584 3018 raft_consensus.cc:2229] T 75b67ccb9a334b25ae71b553a2a84a02 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:40.091730 3018 raft_consensus.cc:2258] T 75b67ccb9a334b25ae71b553a2a84a02 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:40.092666 3018 ts_tablet_manager.cc:1875] T 75b67ccb9a334b25ae71b553a2a84a02 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:40.094897 3018 ts_tablet_manager.cc:1888] T 75b67ccb9a334b25ae71b553a2a84a02 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:40.095196 3018 log.cc:1196] T 75b67ccb9a334b25ae71b553a2a84a02 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/75b67ccb9a334b25ae71b553a2a84a02
I20240501 14:09:40.095801 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 75b67ccb9a334b25ae71b553a2a84a02 (table rebalance_test_table_19 [id=00bf2d6ad5af4e6480096d5ceb4b1bb8]) successfully deleted
I20240501 14:09:40.119607 2960 consensus_queue.cc:1035] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.123986 2960 consensus_queue.cc:1035] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.167403 29873 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:40.173139 348 raft_consensus.cc:1261] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:40.173651 3042 consensus_queue.cc:1035] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.175005 30264 raft_consensus.cc:1261] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:40.176496 29873 catalog_manager.cc:5576] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937) does not have the latest schema for tablet 66ca98d099e24f0f86d4eca60011e107 (table rebalancer_dynamic_table [id=ab5ec2915fc044ad9585146a2afdd70e]). Expected version 1 got 0
I20240501 14:09:40.177064 2421 consensus_queue.cc:1035] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.178959 2946 tablet.cc:1722] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:40.179270 2946 tablet.cc:1620] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:40.182098 29873 catalog_manager.cc:5576] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103) does not have the latest schema for tablet 66ca98d099e24f0f86d4eca60011e107 (table rebalancer_dynamic_table [id=ab5ec2915fc044ad9585146a2afdd70e]). Expected version 1 got 0
I20240501 14:09:40.183441 29871 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=ab5ec2915fc044ad9585146a2afdd70e] alter complete (version 1)
I20240501 14:09:40.184486 3119 tablet.cc:1722] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:40.184765 3119 tablet.cc:1620] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:40.193326 32534 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:40.195612 3124 tablet.cc:1722] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:40.198060 29873 catalog_manager.cc:5576] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103) does not have the latest schema for tablet 66ca98d099e24f0f86d4eca60011e107 (table rebalancer_dynamic_table [id=ab5ec2915fc044ad9585146a2afdd70e]). Expected version 1 got 0
I20240501 14:09:40.200728 3124 tablet.cc:1620] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a: MemRowSet was empty: no flush needed.
I20240501 14:09:40.202724 30387 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:40.207455 32664 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:40.211908 30124 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:40.214915 32402 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:40.214980 328 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:40.215301 29993 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:40.215544 30256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:40.224148 2203 consensus_queue.cc:1035] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.227968 2203 consensus_queue.cc:1035] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.284144 30145 consensus_queue.cc:237] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:40.285392 32553 raft_consensus.cc:1261] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:40.285742 30264 raft_consensus.cc:1261] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:40.285763 2969 consensus_queue.cc:1035] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:40.286136 2960 consensus_queue.cc:1035] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:40.287564 2960 raft_consensus.cc:2932] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:40.287860 32553 raft_consensus.cc:2932] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:40.288861 30264 raft_consensus.cc:2932] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:40.290436 29873 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:40.290573 29871 catalog_manager.cc:5484] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 reported cstate change: config changed from index 4 to 5, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:40.294585 30387 tablet_service.cc:1514] Processing DeleteTablet for tablet 5518112449d9448e98f5c779f1fa3248 with delete_type TABLET_DATA_TOMBSTONED (TS d1c217469ae4478ebd49e4ed9f7352d7 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:40.294889 3151 tablet_replica.cc:330] T 5518112449d9448e98f5c779f1fa3248 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:40.295076 3151 raft_consensus.cc:2229] T 5518112449d9448e98f5c779f1fa3248 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:40.295240 3151 raft_consensus.cc:2258] T 5518112449d9448e98f5c779f1fa3248 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:40.295771 3151 ts_tablet_manager.cc:1875] T 5518112449d9448e98f5c779f1fa3248 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:40.298133 3151 ts_tablet_manager.cc:1888] T 5518112449d9448e98f5c779f1fa3248 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:40.298283 3151 log.cc:1196] T 5518112449d9448e98f5c779f1fa3248 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/5518112449d9448e98f5c779f1fa3248
I20240501 14:09:40.298596 3119 tablet.cc:1722] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:40.298965 3119 tablet.cc:1620] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:40.298960 3124 tablet.cc:1722] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:40.299248 3124 tablet.cc:1620] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a: MemRowSet was empty: no flush needed.
I20240501 14:09:40.299458 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 5518112449d9448e98f5c779f1fa3248 (table rebalance_test_table_5 [id=440878c63b0f4706b34fd42316991270]) successfully deleted
I20240501 14:09:40.302680 2946 tablet.cc:1722] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:40.303006 2946 tablet.cc:1620] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:40.304410 29873 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=ab5ec2915fc044ad9585146a2afdd70e] alter complete (version 2)
I20240501 14:09:40.378113 32534 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:40.382460 328 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:40.384851 32402 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:40.390527 30124 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:40.392786 30256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:40.396004 32664 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:40.401443 30387 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:40.406746 29866 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:40.412681 3119 tablet.cc:1722] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:40.416272 2946 tablet.cc:1722] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:40.417479 29871 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=ab5ec2915fc044ad9585146a2afdd70e] alter complete (version 3)
I20240501 14:09:40.421849 3124 tablet.cc:1722] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:40.431808 29993 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:40.470932 32415 consensus_queue.cc:237] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:40.472110 349 raft_consensus.cc:1261] T 35dd2c9d436448c482ef0b3fac3a3b64 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:40.472199 30406 raft_consensus.cc:1261] T 35dd2c9d436448c482ef0b3fac3a3b64 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:40.472592 2944 consensus_queue.cc:1035] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:40.472767 2503 consensus_queue.cc:1035] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:40.474422 2944 raft_consensus.cc:2932] T 35dd2c9d436448c482ef0b3fac3a3b64 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:40.474857 30406 raft_consensus.cc:2932] T 35dd2c9d436448c482ef0b3fac3a3b64 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:40.475100 349 raft_consensus.cc:2932] T 35dd2c9d436448c482ef0b3fac3a3b64 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:40.478022 29871 catalog_manager.cc:5484] T 35dd2c9d436448c482ef0b3fac3a3b64 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:40.480966 32534 tablet_service.cc:1514] Processing DeleteTablet for tablet 35dd2c9d436448c482ef0b3fac3a3b64 with delete_type TABLET_DATA_TOMBSTONED (TS 5f8a0cfdfdca4148a75686dc88059cf3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:40.481293 3223 tablet_replica.cc:330] T 35dd2c9d436448c482ef0b3fac3a3b64 P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:40.481452 3223 raft_consensus.cc:2229] T 35dd2c9d436448c482ef0b3fac3a3b64 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:40.481575 3223 raft_consensus.cc:2258] T 35dd2c9d436448c482ef0b3fac3a3b64 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:40.482021 3223 ts_tablet_manager.cc:1875] T 35dd2c9d436448c482ef0b3fac3a3b64 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:40.484351 3223 ts_tablet_manager.cc:1888] T 35dd2c9d436448c482ef0b3fac3a3b64 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:09:40.484433 3223 log.cc:1196] T 35dd2c9d436448c482ef0b3fac3a3b64 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/35dd2c9d436448c482ef0b3fac3a3b64
I20240501 14:09:40.485272 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet 35dd2c9d436448c482ef0b3fac3a3b64 (table rebalancer_extra_table_12 [id=dcad19a5572a46a992d9c7fd61cfcf2b]) successfully deleted
I20240501 14:09:40.510046 29867 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:40.516023 2503 consensus_queue.cc:1035] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.517194 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 081a6dc5b80c49ce8807c11eaca0439b (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=a1c79c761a9c49e69c832cdf99272f4c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.517490 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 081a6dc5b80c49ce8807c11eaca0439b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.518000 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 8f4669cad44f4a3c9e01c4bba3c14734 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=a1c79c761a9c49e69c832cdf99272f4c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.518292 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f4669cad44f4a3c9e01c4bba3c14734. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.519748 32402 tablet_service.cc:1467] Processing CreateTablet for tablet 081a6dc5b80c49ce8807c11eaca0439b (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=a1c79c761a9c49e69c832cdf99272f4c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.520031 32402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 081a6dc5b80c49ce8807c11eaca0439b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.519968 30387 tablet_service.cc:1467] Processing CreateTablet for tablet 081a6dc5b80c49ce8807c11eaca0439b (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=a1c79c761a9c49e69c832cdf99272f4c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.520336 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 081a6dc5b80c49ce8807c11eaca0439b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.520995 328 tablet_service.cc:1467] Processing CreateTablet for tablet 427147f3c9934f87805d3730abaf5d63 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=a1c79c761a9c49e69c832cdf99272f4c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.521283 328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 427147f3c9934f87805d3730abaf5d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.519748 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 427147f3c9934f87805d3730abaf5d63 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=a1c79c761a9c49e69c832cdf99272f4c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.522140 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 427147f3c9934f87805d3730abaf5d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.522512 3243 tablet_bootstrap.cc:492] T 081a6dc5b80c49ce8807c11eaca0439b P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:40.522707 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 427147f3c9934f87805d3730abaf5d63 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=a1c79c761a9c49e69c832cdf99272f4c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.522917 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 427147f3c9934f87805d3730abaf5d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.523419 3243 tablet_bootstrap.cc:654] T 081a6dc5b80c49ce8807c11eaca0439b P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.524711 2967 tablet_bootstrap.cc:492] T 081a6dc5b80c49ce8807c11eaca0439b P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:40.521899 3242 tablet_bootstrap.cc:492] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:40.524976 3243 tablet_bootstrap.cc:492] T 081a6dc5b80c49ce8807c11eaca0439b P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:40.525058 3243 ts_tablet_manager.cc:1367] T 081a6dc5b80c49ce8807c11eaca0439b P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:40.525353 29866 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:40.525457 29866 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:40.525645 2967 tablet_bootstrap.cc:654] T 081a6dc5b80c49ce8807c11eaca0439b P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.525815 3242 tablet_bootstrap.cc:654] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.526540 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 8f4669cad44f4a3c9e01c4bba3c14734 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=a1c79c761a9c49e69c832cdf99272f4c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.526827 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f4669cad44f4a3c9e01c4bba3c14734. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.527798 29866 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 66ca98d099e24f0f86d4eca60011e107
I20240501 14:09:40.528625 30256 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ca98d099e24f0f86d4eca60011e107 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:40 UTC) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:40.528757 328 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ca98d099e24f0f86d4eca60011e107 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:40 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:40.528888 32665 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ca98d099e24f0f86d4eca60011e107 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:40 UTC) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:40.529049 3249 tablet_replica.cc:330] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:40.529268 3248 tablet_replica.cc:330] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:40.529348 3249 raft_consensus.cc:2229] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:40.529410 3248 raft_consensus.cc:2229] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:40.529464 3249 raft_consensus.cc:2258] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:40.529520 3248 raft_consensus.cc:2258] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:40.529907 2503 consensus_queue.cc:1035] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.530105 3249 ts_tablet_manager.cc:1875] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:40.530331 30124 tablet_service.cc:1467] Processing CreateTablet for tablet 8f4669cad44f4a3c9e01c4bba3c14734 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=a1c79c761a9c49e69c832cdf99272f4c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.530568 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f4669cad44f4a3c9e01c4bba3c14734. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.533162 2967 tablet_bootstrap.cc:492] T 081a6dc5b80c49ce8807c11eaca0439b P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:40.533259 2967 ts_tablet_manager.cc:1367] T 081a6dc5b80c49ce8807c11eaca0439b P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:40.533337 3249 ts_tablet_manager.cc:1888] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:40.533399 3249 log.cc:1196] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/66ca98d099e24f0f86d4eca60011e107
I20240501 14:09:40.533648 3249 ts_tablet_manager.cc:1909] T 66ca98d099e24f0f86d4eca60011e107 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:09:40.533732 2967 raft_consensus.cc:348] T 081a6dc5b80c49ce8807c11eaca0439b P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:40.533830 2967 raft_consensus.cc:374] T 081a6dc5b80c49ce8807c11eaca0439b P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.533860 2967 raft_consensus.cc:729] T 081a6dc5b80c49ce8807c11eaca0439b P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.533977 2967 consensus_queue.cc:260] T 081a6dc5b80c49ce8807c11eaca0439b P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:40.534257 2967 ts_tablet_manager.cc:1398] T 081a6dc5b80c49ce8807c11eaca0439b P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.534449 2967 tablet_bootstrap.cc:492] T 8f4669cad44f4a3c9e01c4bba3c14734 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:40.535341 2967 tablet_bootstrap.cc:654] T 8f4669cad44f4a3c9e01c4bba3c14734 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.535338 3243 raft_consensus.cc:348] T 081a6dc5b80c49ce8807c11eaca0439b P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:40.535445 3243 raft_consensus.cc:374] T 081a6dc5b80c49ce8807c11eaca0439b P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.535498 3243 raft_consensus.cc:729] T 081a6dc5b80c49ce8807c11eaca0439b P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.535614 3243 consensus_queue.cc:260] T 081a6dc5b80c49ce8807c11eaca0439b P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:40.535868 3243 ts_tablet_manager.cc:1398] T 081a6dc5b80c49ce8807c11eaca0439b P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240501 14:09:40.536051 3243 tablet_bootstrap.cc:492] T 427147f3c9934f87805d3730abaf5d63 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:40.536710 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 66ca98d099e24f0f86d4eca60011e107 (table rebalancer_dynamic_table_new_name [id=ab5ec2915fc044ad9585146a2afdd70e]) successfully deleted
I20240501 14:09:40.537108 3243 tablet_bootstrap.cc:654] T 427147f3c9934f87805d3730abaf5d63 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.537298 3274 tablet_bootstrap.cc:492] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:40.538218 3274 tablet_bootstrap.cc:654] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.539713 3274 tablet_bootstrap.cc:492] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:40.539770 3244 tablet_bootstrap.cc:492] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:40.539800 3274 ts_tablet_manager.cc:1367] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:40.540184 3274 raft_consensus.cc:348] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:40.540318 3274 raft_consensus.cc:374] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.540356 3274 raft_consensus.cc:729] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.540479 3274 consensus_queue.cc:260] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:40.540741 3274 ts_tablet_manager.cc:1398] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.540869 3250 tablet_bootstrap.cc:492] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:40.540983 3244 tablet_bootstrap.cc:654] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.542044 2967 tablet_bootstrap.cc:492] T 8f4669cad44f4a3c9e01c4bba3c14734 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:40.542140 2967 ts_tablet_manager.cc:1367] T 8f4669cad44f4a3c9e01c4bba3c14734 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:40.542311 3242 tablet_bootstrap.cc:492] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:40.542398 3242 ts_tablet_manager.cc:1367] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.000s
I20240501 14:09:40.542613 2967 raft_consensus.cc:348] T 8f4669cad44f4a3c9e01c4bba3c14734 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:40.542825 3242 raft_consensus.cc:348] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:40.542910 3242 raft_consensus.cc:374] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.542943 3242 raft_consensus.cc:729] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.543048 3242 consensus_queue.cc:260] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:40.543340 3242 ts_tablet_manager.cc:1398] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.543498 3242 tablet_bootstrap.cc:492] T 427147f3c9934f87805d3730abaf5d63 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:40.543802 2967 raft_consensus.cc:374] T 8f4669cad44f4a3c9e01c4bba3c14734 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.543850 2967 raft_consensus.cc:729] T 8f4669cad44f4a3c9e01c4bba3c14734 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.543960 2967 consensus_queue.cc:260] T 8f4669cad44f4a3c9e01c4bba3c14734 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:40.544262 2967 ts_tablet_manager.cc:1398] T 8f4669cad44f4a3c9e01c4bba3c14734 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:40.545174 3250 tablet_bootstrap.cc:654] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.546801 3251 tablet_replica.cc:330] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:40.546969 3251 raft_consensus.cc:2229] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:40.547428 3250 tablet_bootstrap.cc:492] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:40.548136 3250 ts_tablet_manager.cc:1367] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:40.548449 3248 ts_tablet_manager.cc:1875] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:40.548668 3242 tablet_bootstrap.cc:654] T 427147f3c9934f87805d3730abaf5d63 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.549254 3250 raft_consensus.cc:348] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:40.549407 3250 raft_consensus.cc:374] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.549485 3250 raft_consensus.cc:729] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.550822 3248 ts_tablet_manager.cc:1888] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:40.550899 3248 log.cc:1196] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/66ca98d099e24f0f86d4eca60011e107
I20240501 14:09:40.551532 3248 ts_tablet_manager.cc:1909] T 66ca98d099e24f0f86d4eca60011e107 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting consensus metadata
I20240501 14:09:40.551533 3244 tablet_bootstrap.cc:492] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:40.551617 3244 ts_tablet_manager.cc:1367] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:40.551801 3243 tablet_bootstrap.cc:492] T 427147f3c9934f87805d3730abaf5d63 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:40.551891 3243 ts_tablet_manager.cc:1367] T 427147f3c9934f87805d3730abaf5d63 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:09:40.552109 3244 raft_consensus.cc:348] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:40.552212 3244 raft_consensus.cc:374] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.552268 3244 raft_consensus.cc:729] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.552372 3243 raft_consensus.cc:348] T 427147f3c9934f87805d3730abaf5d63 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:40.552397 3244 consensus_queue.cc:260] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:40.552479 3243 raft_consensus.cc:374] T 427147f3c9934f87805d3730abaf5d63 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.552512 3243 raft_consensus.cc:729] T 427147f3c9934f87805d3730abaf5d63 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.552613 3243 consensus_queue.cc:260] T 427147f3c9934f87805d3730abaf5d63 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:40.552675 3244 ts_tablet_manager.cc:1398] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.552745 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet 66ca98d099e24f0f86d4eca60011e107 (table rebalancer_dynamic_table_new_name [id=ab5ec2915fc044ad9585146a2afdd70e]) successfully deleted
I20240501 14:09:40.552861 3243 ts_tablet_manager.cc:1398] T 427147f3c9934f87805d3730abaf5d63 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.552990 3242 tablet_bootstrap.cc:492] T 427147f3c9934f87805d3730abaf5d63 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:40.553066 3242 ts_tablet_manager.cc:1367] T 427147f3c9934f87805d3730abaf5d63 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 14:09:40.553506 3242 raft_consensus.cc:348] T 427147f3c9934f87805d3730abaf5d63 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:40.553606 3242 raft_consensus.cc:374] T 427147f3c9934f87805d3730abaf5d63 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.553634 3242 raft_consensus.cc:729] T 427147f3c9934f87805d3730abaf5d63 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.553761 3242 consensus_queue.cc:260] T 427147f3c9934f87805d3730abaf5d63 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:40.554072 3242 ts_tablet_manager.cc:1398] T 427147f3c9934f87805d3730abaf5d63 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.556404 3250 consensus_queue.cc:260] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:40.556802 3250 ts_tablet_manager.cc:1398] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240501 14:09:40.557096 3251 raft_consensus.cc:2258] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:40.558327 3251 ts_tablet_manager.cc:1875] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:40.560559 3251 ts_tablet_manager.cc:1888] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:40.560684 3251 log.cc:1196] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/66ca98d099e24f0f86d4eca60011e107
I20240501 14:09:40.561014 3251 ts_tablet_manager.cc:1909] T 66ca98d099e24f0f86d4eca60011e107 P db935501e9104de5b15c8780681d6c38: Deleting consensus metadata
I20240501 14:09:40.561566 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet 66ca98d099e24f0f86d4eca60011e107 (table rebalancer_dynamic_table_new_name [id=ab5ec2915fc044ad9585146a2afdd70e]) successfully deleted
I20240501 14:09:40.587898 3126 raft_consensus.cc:482] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:40.588012 3126 raft_consensus.cc:504] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:40.588395 3126 leader_election.cc:288] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:40.588732 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "427147f3c9934f87805d3730abaf5d63" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:40.588850 32415 raft_consensus.cc:2454] T 427147f3c9934f87805d3730abaf5d63 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:40.589264 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "427147f3c9934f87805d3730abaf5d63" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:40.589387 30406 raft_consensus.cc:2454] T 427147f3c9934f87805d3730abaf5d63 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:40.589386 32752 leader_election.cc:302] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:40.589704 3126 raft_consensus.cc:2781] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:40.589758 3126 raft_consensus.cc:482] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:40.589793 3126 raft_consensus.cc:3037] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.590764 3126 raft_consensus.cc:504] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:40.591071 3126 leader_election.cc:288] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:40.591328 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "427147f3c9934f87805d3730abaf5d63" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:40.591426 30406 raft_consensus.cc:3037] T 427147f3c9934f87805d3730abaf5d63 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.591557 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "427147f3c9934f87805d3730abaf5d63" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:40.591650 32415 raft_consensus.cc:3037] T 427147f3c9934f87805d3730abaf5d63 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.592398 30406 raft_consensus.cc:2454] T 427147f3c9934f87805d3730abaf5d63 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:40.592528 32415 raft_consensus.cc:2454] T 427147f3c9934f87805d3730abaf5d63 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:40.592856 32752 leader_election.cc:302] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:40.593178 3126 raft_consensus.cc:2781] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:40.593259 3126 raft_consensus.cc:686] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:40.593364 3126 consensus_queue.cc:237] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:40.595127 29873 catalog_manager.cc:5484] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:40.633451 29871 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:40.640440 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 9c78448e8b0940dab91e602feab79a50 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=59fce4bc40f0466c9848c75337165a4a]), partition=
I20240501 14:09:40.640591 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 9c78448e8b0940dab91e602feab79a50 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=59fce4bc40f0466c9848c75337165a4a]), partition=
I20240501 14:09:40.640703 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c78448e8b0940dab91e602feab79a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.640843 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c78448e8b0940dab91e602feab79a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.642858 2967 tablet_bootstrap.cc:492] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:40.642971 3243 tablet_bootstrap.cc:492] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:40.643759 2967 tablet_bootstrap.cc:654] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.643822 3243 tablet_bootstrap.cc:654] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.644491 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 9c78448e8b0940dab91e602feab79a50 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=59fce4bc40f0466c9848c75337165a4a]), partition=
I20240501 14:09:40.644737 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c78448e8b0940dab91e602feab79a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.646524 3243 tablet_bootstrap.cc:492] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:40.646605 3243 ts_tablet_manager.cc:1367] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:40.647065 3243 raft_consensus.cc:348] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:40.647186 3243 raft_consensus.cc:374] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.647222 3243 raft_consensus.cc:729] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.647361 3243 consensus_queue.cc:260] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:40.647643 3243 ts_tablet_manager.cc:1398] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.652845 2967 tablet_bootstrap.cc:492] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:40.652935 2967 ts_tablet_manager.cc:1367] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240501 14:09:40.653101 3338 tablet_bootstrap.cc:492] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:40.653378 2967 raft_consensus.cc:348] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:40.653476 2967 raft_consensus.cc:374] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.653509 2967 raft_consensus.cc:729] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.653641 2967 consensus_queue.cc:260] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:40.653924 2967 ts_tablet_manager.cc:1398] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:40.654300 3338 tablet_bootstrap.cc:654] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.655597 3338 tablet_bootstrap.cc:492] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:40.655740 3338 ts_tablet_manager.cc:1367] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:40.656518 3338 raft_consensus.cc:348] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:40.656656 3338 raft_consensus.cc:374] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.656711 3338 raft_consensus.cc:729] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.656867 3338 consensus_queue.cc:260] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:40.657207 3338 ts_tablet_manager.cc:1398] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.726365 2914 raft_consensus.cc:482] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:40.726467 2914 raft_consensus.cc:504] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:40.726845 2914 leader_election.cc:288] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:40.727137 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4669cad44f4a3c9e01c4bba3c14734" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:40.727134 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4669cad44f4a3c9e01c4bba3c14734" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:40.727255 30145 raft_consensus.cc:2454] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:40.727267 32685 raft_consensus.cc:2454] T 8f4669cad44f4a3c9e01c4bba3c14734 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:40.727517 32487 leader_election.cc:302] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:40.727664 2914 raft_consensus.cc:2781] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:40.727723 2914 raft_consensus.cc:482] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:40.727762 2914 raft_consensus.cc:3037] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.728672 2914 raft_consensus.cc:504] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:40.728983 2914 leader_election.cc:288] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:40.729256 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4669cad44f4a3c9e01c4bba3c14734" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:40.729274 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4669cad44f4a3c9e01c4bba3c14734" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:40.729357 30145 raft_consensus.cc:3037] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.729368 32685 raft_consensus.cc:3037] T 8f4669cad44f4a3c9e01c4bba3c14734 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.730113 30145 raft_consensus.cc:2454] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:40.730252 32685 raft_consensus.cc:2454] T 8f4669cad44f4a3c9e01c4bba3c14734 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:40.730369 32490 leader_election.cc:302] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:40.730530 2914 raft_consensus.cc:2781] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:40.730695 2914 raft_consensus.cc:686] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:09:40.730866 2914 consensus_queue.cc:237] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:40.732896 29866 catalog_manager.cc:5484] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:40.793169 2503 raft_consensus.cc:482] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:40.793296 2503 raft_consensus.cc:504] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:40.793653 2503 leader_election.cc:288] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:40.793967 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081a6dc5b80c49ce8807c11eaca0439b" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:40.794095 32685 raft_consensus.cc:2454] T 081a6dc5b80c49ce8807c11eaca0439b P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:40.794066 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081a6dc5b80c49ce8807c11eaca0439b" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:40.794198 30406 raft_consensus.cc:2454] T 081a6dc5b80c49ce8807c11eaca0439b P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:40.794377 32356 leader_election.cc:302] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:40.794615 2503 raft_consensus.cc:2781] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:40.794673 2503 raft_consensus.cc:482] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:40.794714 2503 raft_consensus.cc:3037] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.795771 2503 raft_consensus.cc:504] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:40.796073 2503 leader_election.cc:288] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:40.796489 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081a6dc5b80c49ce8807c11eaca0439b" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:40.796572 32685 raft_consensus.cc:3037] T 081a6dc5b80c49ce8807c11eaca0439b P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.796823 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081a6dc5b80c49ce8807c11eaca0439b" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:40.796911 30406 raft_consensus.cc:3037] T 081a6dc5b80c49ce8807c11eaca0439b P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.797647 32685 raft_consensus.cc:2454] T 081a6dc5b80c49ce8807c11eaca0439b P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:40.797854 30406 raft_consensus.cc:2454] T 081a6dc5b80c49ce8807c11eaca0439b P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:40.797967 32356 leader_election.cc:302] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:40.798206 2944 raft_consensus.cc:2781] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:40.798297 2944 raft_consensus.cc:686] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:40.798409 2944 consensus_queue.cc:237] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:40.800798 29866 catalog_manager.cc:5484] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:40.812181 3017 raft_consensus.cc:482] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:40.812323 3017 raft_consensus.cc:504] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:40.812717 3017 leader_election.cc:288] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:40.813413 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c78448e8b0940dab91e602feab79a50" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:40.813525 30406 raft_consensus.cc:2454] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:40.815419 29948 leader_election.cc:302] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:40.815570 3017 raft_consensus.cc:2781] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:40.815637 3017 raft_consensus.cc:482] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:40.815670 3017 raft_consensus.cc:3037] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.816180 30256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:40.816711 3017 raft_consensus.cc:504] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:40.817021 3017 leader_election.cc:288] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:40.818712 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c78448e8b0940dab91e602feab79a50" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:40.818821 30406 raft_consensus.cc:3037] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.819818 30406 raft_consensus.cc:2454] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:40.820114 29948 leader_election.cc:302] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:40.820283 3017 raft_consensus.cc:2781] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:40.820410 3017 raft_consensus.cc:686] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:40.820526 3017 consensus_queue.cc:237] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:40.823230 29866 catalog_manager.cc:5484] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:40.825194 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c78448e8b0940dab91e602feab79a50" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:40.825322 32685 raft_consensus.cc:2454] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:40.825475 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c78448e8b0940dab91e602feab79a50" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:40.825542 32683 raft_consensus.cc:3037] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.826448 32683 raft_consensus.cc:2454] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:40.827389 32534 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:40.836766 328 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:40.838286 30124 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:40.847544 32400 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:40.848469 32665 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:40.849500 30388 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:40.860533 29993 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:40.906136 349 consensus_queue.cc:237] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:40.907313 32683 raft_consensus.cc:1261] T f2541122e05f45c2843d06c5342c8d01 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:40.907326 32415 raft_consensus.cc:1261] T f2541122e05f45c2843d06c5342c8d01 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:40.907707 3376 consensus_queue.cc:1035] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:40.907871 3126 consensus_queue.cc:1035] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:40.909651 2203 raft_consensus.cc:2932] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:40.910111 32683 raft_consensus.cc:2932] T f2541122e05f45c2843d06c5342c8d01 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:40.911527 32415 raft_consensus.cc:2932] T f2541122e05f45c2843d06c5342c8d01 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:40.912945 29866 catalog_manager.cc:5484] T f2541122e05f45c2843d06c5342c8d01 P db935501e9104de5b15c8780681d6c38 reported cstate change: config changed from index 2 to 3, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:40.915815 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet f2541122e05f45c2843d06c5342c8d01 with delete_type TABLET_DATA_TOMBSTONED (TS 93f2518fbadf4f9991cda93afacb363e not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:40.916177 3384 tablet_replica.cc:330] T f2541122e05f45c2843d06c5342c8d01 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:40.916338 3384 raft_consensus.cc:2229] T f2541122e05f45c2843d06c5342c8d01 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:40.916487 3384 raft_consensus.cc:2258] T f2541122e05f45c2843d06c5342c8d01 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:40.916975 3384 ts_tablet_manager.cc:1875] T f2541122e05f45c2843d06c5342c8d01 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:40.919209 3384 ts_tablet_manager.cc:1888] T f2541122e05f45c2843d06c5342c8d01 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:09:40.919271 3384 log.cc:1196] T f2541122e05f45c2843d06c5342c8d01 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/f2541122e05f45c2843d06c5342c8d01
I20240501 14:09:40.919791 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet f2541122e05f45c2843d06c5342c8d01 (table rebalancer_extra_table_6 [id=d23b9f10156c4074a707699cf10e3b55]) successfully deleted
I20240501 14:09:40.937826 29871 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:40.941892 30406 raft_consensus.cc:1261] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:40.941962 32683 raft_consensus.cc:1261] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:40.942553 3358 consensus_queue.cc:1035] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.942881 3358 consensus_queue.cc:1035] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.945701 29871 catalog_manager.cc:5576] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357) does not have the latest schema for tablet 9c78448e8b0940dab91e602feab79a50 (table rebalancer_dynamic_table [id=59fce4bc40f0466c9848c75337165a4a]). Expected version 1 got 0
I20240501 14:09:40.946210 29866 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet 9c78448e8b0940dab91e602feab79a50 (table rebalancer_dynamic_table [id=59fce4bc40f0466c9848c75337165a4a]). Expected version 1 got 0
I20240501 14:09:40.947433 3401 tablet.cc:1722] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:40.947657 3400 tablet.cc:1722] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:40.948022 3400 tablet.cc:1620] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e: MemRowSet was empty: no flush needed.
I20240501 14:09:40.948022 3401 tablet.cc:1620] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:40.949259 29866 catalog_manager.cc:5576] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357) does not have the latest schema for tablet 9c78448e8b0940dab91e602feab79a50 (table rebalancer_dynamic_table [id=59fce4bc40f0466c9848c75337165a4a]). Expected version 1 got 0
I20240501 14:09:40.950047 3402 tablet.cc:1722] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:40.950124 29871 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet 9c78448e8b0940dab91e602feab79a50 (table rebalancer_dynamic_table [id=59fce4bc40f0466c9848c75337165a4a]). Expected version 1 got 0
I20240501 14:09:40.950366 3402 tablet.cc:1620] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7: MemRowSet was empty: no flush needed.
I20240501 14:09:40.953696 29866 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=59fce4bc40f0466c9848c75337165a4a] alter complete (version 1)
I20240501 14:09:41.019618 3376 consensus_queue.cc:1035] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:41.025738 32534 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:41.026938 2203 consensus_queue.cc:1035] T 427147f3c9934f87805d3730abaf5d63 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:41.027325 30256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:41.029021 29993 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:41.033708 30124 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:41.039736 32400 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:41.045130 30388 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:41.056452 29864 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:41.062239 32665 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:41.062456 3400 tablet.cc:1722] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:41.062793 3400 tablet.cc:1620] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e: MemRowSet was empty: no flush needed.
I20240501 14:09:41.063822 3401 tablet.cc:1722] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:41.064085 3401 tablet.cc:1620] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:41.064343 3402 tablet.cc:1722] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:41.064597 3402 tablet.cc:1620] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7: MemRowSet was empty: no flush needed.
I20240501 14:09:41.073175 29864 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=59fce4bc40f0466c9848c75337165a4a] alter complete (version 2)
I20240501 14:09:41.083010 328 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:41.121824 30145 consensus_queue.cc:237] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.123181 349 raft_consensus.cc:1261] T 3b6a526c2ed84e1780ec392ef39d09b0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:41.123289 30406 raft_consensus.cc:1261] T 3b6a526c2ed84e1780ec392ef39d09b0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:41.123589 2960 consensus_queue.cc:1035] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:41.123778 2969 consensus_queue.cc:1035] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:41.125614 2969 raft_consensus.cc:2932] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:41.126042 30406 raft_consensus.cc:2932] T 3b6a526c2ed84e1780ec392ef39d09b0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:41.126309 349 raft_consensus.cc:2932] T 3b6a526c2ed84e1780ec392ef39d09b0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:41.128998 29870 catalog_manager.cc:5484] T 3b6a526c2ed84e1780ec392ef39d09b0 P d35c16dcb506426993811bb2519d7db3 reported cstate change: config changed from index 4 to 6, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New cstate: current_term: 2 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:41.131834 30256 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b6a526c2ed84e1780ec392ef39d09b0 with delete_type TABLET_DATA_TOMBSTONED (TS 1d0bb69611af4876a2b3b33c4766e13a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:41.132166 3475 tablet_replica.cc:330] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:41.132340 3475 raft_consensus.cc:2229] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:41.132441 3475 raft_consensus.cc:2258] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:41.132915 3475 ts_tablet_manager.cc:1875] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:41.135586 3475 ts_tablet_manager.cc:1888] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 14:09:41.135679 3475 log.cc:1196] T 3b6a526c2ed84e1780ec392ef39d09b0 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/3b6a526c2ed84e1780ec392ef39d09b0
I20240501 14:09:41.136265 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet 3b6a526c2ed84e1780ec392ef39d09b0 (table rebalance_test_table_19 [id=00bf2d6ad5af4e6480096d5ceb4b1bb8]) successfully deleted
I20240501 14:09:41.174466 29872 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:41.178800 2914 consensus_queue.cc:1035] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:41.180655 3400 tablet.cc:1722] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:41.181533 3402 tablet.cc:1722] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:41.186584 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=59fce4bc40f0466c9848c75337165a4a] alter complete (version 3)
I20240501 14:09:41.190474 2914 consensus_queue.cc:1035] T 8f4669cad44f4a3c9e01c4bba3c14734 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:41.190624 30145 consensus_queue.cc:237] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:41.191540 3401 tablet.cc:1722] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:41.193259 30264 raft_consensus.cc:1261] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:41.193678 2944 consensus_queue.cc:1035] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:41.194823 30406 raft_consensus.cc:1261] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:41.194779 3516 consensus_queue.cc:1035] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:41.195278 3516 consensus_queue.cc:1035] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:41.196031 30079 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c362bd01a59b4bcf85eeb59713e58b64. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:41.199429 2503 consensus_queue.cc:1035] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:41.205067 2960 raft_consensus.cc:2932] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:41.206492 30264 raft_consensus.cc:2932] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:41.207916 29872 catalog_manager.cc:5484] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 reported cstate change: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:41.209153 30406 raft_consensus.cc:2932] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:41.230334 30146 consensus_queue.cc:237] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.231621 30264 raft_consensus.cc:1261] T ee998c3fcf3449d5845ac1c950f326d7 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:41.231664 30013 raft_consensus.cc:1261] T ee998c3fcf3449d5845ac1c950f326d7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:41.231719 30081 consensus_peers.cc:485] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 -> Peer 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): Couldn't send request to peer 9ecd907cf5634ac98f3f54f3dbec6849. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee998c3fcf3449d5845ac1c950f326d7. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:41.231941 3516 consensus_queue.cc:1035] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:41.232112 2969 consensus_queue.cc:1035] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:41.233973 2969 raft_consensus.cc:2932] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:41.234432 30013 raft_consensus.cc:2932] T ee998c3fcf3449d5845ac1c950f326d7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:41.235879 30264 raft_consensus.cc:2932] T ee998c3fcf3449d5845ac1c950f326d7 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:41.238003 29872 catalog_manager.cc:5484] T ee998c3fcf3449d5845ac1c950f326d7 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:41.256546 30264 consensus_queue.cc:237] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:41.257815 30013 raft_consensus.cc:1261] T 4a5869fb581b4939ad2207ab9200d674 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:41.257972 30212 consensus_peers.cc:485] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): Couldn't send request to peer a91029b90bf9415a89a2f70736759ad3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a5869fb581b4939ad2207ab9200d674. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:41.258142 30146 raft_consensus.cc:1261] T 4a5869fb581b4939ad2207ab9200d674 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:41.258178 3557 consensus_queue.cc:1035] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:41.258538 2956 consensus_queue.cc:1035] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:41.260304 3557 raft_consensus.cc:2932] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:41.260704 30146 raft_consensus.cc:2932] T 4a5869fb581b4939ad2207ab9200d674 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:41.262459 30013 raft_consensus.cc:2932] T 4a5869fb581b4939ad2207ab9200d674 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:41.263422 29870 catalog_manager.cc:5484] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:41.282778 32415 consensus_queue.cc:237] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:41.284087 30406 raft_consensus.cc:1261] T 588dd26e1820409a9dfccd8ea6a30f49 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:41.284099 32685 raft_consensus.cc:1261] T 588dd26e1820409a9dfccd8ea6a30f49 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:41.284574 2944 consensus_queue.cc:1035] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:09:41.284740 32358 consensus_peers.cc:485] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 -> Peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): Couldn't send request to peer 1d0bb69611af4876a2b3b33c4766e13a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 588dd26e1820409a9dfccd8ea6a30f49. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:41.284757 2503 consensus_queue.cc:1035] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:41.286710 2503 raft_consensus.cc:2932] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:41.287138 30406 raft_consensus.cc:2932] T 588dd26e1820409a9dfccd8ea6a30f49 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:41.289474 29872 catalog_manager.cc:5484] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:41.290159 29870 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:41.290273 29870 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:41.291288 32685 raft_consensus.cc:2932] T 588dd26e1820409a9dfccd8ea6a30f49 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:41.293939 29870 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 9c78448e8b0940dab91e602feab79a50
I20240501 14:09:41.294523 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c78448e8b0940dab91e602feab79a50 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:41 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:41.294826 32665 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c78448e8b0940dab91e602feab79a50 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:41 UTC) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:41.295011 3384 tablet_replica.cc:330] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:41.295120 30388 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c78448e8b0940dab91e602feab79a50 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:41 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:41.295164 3384 raft_consensus.cc:2229] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:41.295367 3384 raft_consensus.cc:2258] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:41.295383 3586 tablet_replica.cc:330] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:41.295495 3586 raft_consensus.cc:2229] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:41.295642 3586 raft_consensus.cc:2258] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:41.296054 3384 ts_tablet_manager.cc:1875] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:41.296466 3585 tablet_replica.cc:330] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:41.296590 3585 raft_consensus.cc:2229] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:41.296700 3585 raft_consensus.cc:2258] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:41.296757 3586 ts_tablet_manager.cc:1875] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:41.297293 3585 ts_tablet_manager.cc:1875] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:41.298651 3384 ts_tablet_manager.cc:1888] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:41.298766 3384 log.cc:1196] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/9c78448e8b0940dab91e602feab79a50
I20240501 14:09:41.299068 3384 ts_tablet_manager.cc:1909] T 9c78448e8b0940dab91e602feab79a50 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:41.300257 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 9c78448e8b0940dab91e602feab79a50 (table rebalancer_dynamic_table_new_name [id=59fce4bc40f0466c9848c75337165a4a]) successfully deleted
I20240501 14:09:41.300616 3586 ts_tablet_manager.cc:1888] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:41.300685 3586 log.cc:1196] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/9c78448e8b0940dab91e602feab79a50
I20240501 14:09:41.300940 3586 ts_tablet_manager.cc:1909] T 9c78448e8b0940dab91e602feab79a50 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:41.301403 3585 ts_tablet_manager.cc:1888] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:41.301463 3585 log.cc:1196] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/9c78448e8b0940dab91e602feab79a50
I20240501 14:09:41.301623 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 9c78448e8b0940dab91e602feab79a50 (table rebalancer_dynamic_table_new_name [id=59fce4bc40f0466c9848c75337165a4a]) successfully deleted
I20240501 14:09:41.301710 3585 ts_tablet_manager.cc:1909] T 9c78448e8b0940dab91e602feab79a50 P db935501e9104de5b15c8780681d6c38: Deleting consensus metadata
I20240501 14:09:41.302271 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet 9c78448e8b0940dab91e602feab79a50 (table rebalancer_dynamic_table_new_name [id=59fce4bc40f0466c9848c75337165a4a]) successfully deleted
I20240501 14:09:41.315580 32415 consensus_queue.cc:237] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:41.315802 29866 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:41.317462 32553 raft_consensus.cc:1261] T 776d6722a14345f490f69f90d80d6a6a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:41.317462 348 raft_consensus.cc:1261] T 776d6722a14345f490f69f90d80d6a6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:41.317473 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Couldn't send request to peer d35c16dcb506426993811bb2519d7db3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 776d6722a14345f490f69f90d80d6a6a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:41.317835 2503 consensus_queue.cc:1035] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:41.318336 2503 consensus_queue.cc:1035] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:41.320163 2944 raft_consensus.cc:2932] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:41.320623 32553 raft_consensus.cc:2932] T 776d6722a14345f490f69f90d80d6a6a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:41.322055 348 raft_consensus.cc:2932] T 776d6722a14345f490f69f90d80d6a6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:41.322762 29866 catalog_manager.cc:5484] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:41.324787 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 5db25d87e72347f4a4410bcde440f359 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=0c280593e367426fa4c2534483cb4da4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:41.324858 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 36bf7d2992034055bccfd8146449b696 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=0c280593e367426fa4c2534483cb4da4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:41.325054 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5db25d87e72347f4a4410bcde440f359. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.325134 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36bf7d2992034055bccfd8146449b696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.325615 328 tablet_service.cc:1467] Processing CreateTablet for tablet 5db25d87e72347f4a4410bcde440f359 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=0c280593e367426fa4c2534483cb4da4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:41.325845 328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5db25d87e72347f4a4410bcde440f359. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.326874 3600 tablet_bootstrap.cc:492] T 5db25d87e72347f4a4410bcde440f359 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:41.327047 329 tablet_service.cc:1467] Processing CreateTablet for tablet 36bf7d2992034055bccfd8146449b696 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=0c280593e367426fa4c2534483cb4da4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:41.327268 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36bf7d2992034055bccfd8146449b696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.327476 327 tablet_service.cc:1467] Processing CreateTablet for tablet 90f51696d59c4f11bd44050fde07c276 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=0c280593e367426fa4c2534483cb4da4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:41.327721 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f51696d59c4f11bd44050fde07c276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.328121 3600 tablet_bootstrap.cc:654] T 5db25d87e72347f4a4410bcde440f359 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.328267 30256 tablet_service.cc:1467] Processing CreateTablet for tablet 36bf7d2992034055bccfd8146449b696 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=0c280593e367426fa4c2534483cb4da4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:41.328541 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36bf7d2992034055bccfd8146449b696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.329249 3600 tablet_bootstrap.cc:492] T 5db25d87e72347f4a4410bcde440f359 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:41.329321 3600 ts_tablet_manager.cc:1367] T 5db25d87e72347f4a4410bcde440f359 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:41.329733 3600 raft_consensus.cc:348] T 5db25d87e72347f4a4410bcde440f359 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.329835 3600 raft_consensus.cc:374] T 5db25d87e72347f4a4410bcde440f359 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.329866 3600 raft_consensus.cc:729] T 5db25d87e72347f4a4410bcde440f359 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.329980 3600 consensus_queue.cc:260] T 5db25d87e72347f4a4410bcde440f359 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.324162 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 5db25d87e72347f4a4410bcde440f359 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=0c280593e367426fa4c2534483cb4da4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:41.330224 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5db25d87e72347f4a4410bcde440f359. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.330233 3600 ts_tablet_manager.cc:1398] T 5db25d87e72347f4a4410bcde440f359 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:41.330827 3604 tablet_bootstrap.cc:492] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:41.331089 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 90f51696d59c4f11bd44050fde07c276 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=0c280593e367426fa4c2534483cb4da4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:41.331291 3601 tablet_bootstrap.cc:492] T 36bf7d2992034055bccfd8146449b696 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:41.331403 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f51696d59c4f11bd44050fde07c276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.331717 3604 tablet_bootstrap.cc:654] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.332312 3601 tablet_bootstrap.cc:654] T 36bf7d2992034055bccfd8146449b696 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.335466 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 90f51696d59c4f11bd44050fde07c276 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=0c280593e367426fa4c2534483cb4da4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:41.335539 3606 tablet_bootstrap.cc:492] T 90f51696d59c4f11bd44050fde07c276 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:41.335690 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f51696d59c4f11bd44050fde07c276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.336335 3602 tablet_bootstrap.cc:492] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:41.336758 3606 tablet_bootstrap.cc:654] T 90f51696d59c4f11bd44050fde07c276 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.337492 3601 tablet_bootstrap.cc:492] T 36bf7d2992034055bccfd8146449b696 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:41.337594 3601 ts_tablet_manager.cc:1367] T 36bf7d2992034055bccfd8146449b696 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240501 14:09:41.337934 3600 tablet_bootstrap.cc:492] T 90f51696d59c4f11bd44050fde07c276 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:41.338063 3601 raft_consensus.cc:348] T 36bf7d2992034055bccfd8146449b696 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.338155 3601 raft_consensus.cc:374] T 36bf7d2992034055bccfd8146449b696 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.338186 3601 raft_consensus.cc:729] T 36bf7d2992034055bccfd8146449b696 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.338346 3602 tablet_bootstrap.cc:654] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.338346 3601 consensus_queue.cc:260] T 36bf7d2992034055bccfd8146449b696 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.338469 3606 tablet_bootstrap.cc:492] T 90f51696d59c4f11bd44050fde07c276 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:41.338559 3606 ts_tablet_manager.cc:1367] T 90f51696d59c4f11bd44050fde07c276 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:41.338676 3601 ts_tablet_manager.cc:1398] T 36bf7d2992034055bccfd8146449b696 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:41.338850 3601 tablet_bootstrap.cc:492] T 5db25d87e72347f4a4410bcde440f359 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:41.339321 3600 tablet_bootstrap.cc:654] T 90f51696d59c4f11bd44050fde07c276 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.339509 3604 tablet_bootstrap.cc:492] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:41.339588 3604 ts_tablet_manager.cc:1367] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:41.339903 3601 tablet_bootstrap.cc:654] T 5db25d87e72347f4a4410bcde440f359 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.340031 3604 raft_consensus.cc:348] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.340134 3604 raft_consensus.cc:374] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.340174 3604 raft_consensus.cc:729] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.340307 3604 consensus_queue.cc:260] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.340621 3604 ts_tablet_manager.cc:1398] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:41.341542 3602 tablet_bootstrap.cc:492] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:41.341670 3602 ts_tablet_manager.cc:1367] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:41.341801 3601 tablet_bootstrap.cc:492] T 5db25d87e72347f4a4410bcde440f359 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:41.341902 3601 ts_tablet_manager.cc:1367] T 5db25d87e72347f4a4410bcde440f359 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:41.342267 3602 raft_consensus.cc:348] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.342424 3602 raft_consensus.cc:374] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.342378 3601 raft_consensus.cc:348] T 5db25d87e72347f4a4410bcde440f359 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.342490 3601 raft_consensus.cc:374] T 5db25d87e72347f4a4410bcde440f359 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.342540 3601 raft_consensus.cc:729] T 5db25d87e72347f4a4410bcde440f359 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.342530 3602 raft_consensus.cc:729] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.342654 3601 consensus_queue.cc:260] T 5db25d87e72347f4a4410bcde440f359 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.342737 3600 tablet_bootstrap.cc:492] T 90f51696d59c4f11bd44050fde07c276 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:41.342816 3600 ts_tablet_manager.cc:1367] T 90f51696d59c4f11bd44050fde07c276 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:41.342862 3602 consensus_queue.cc:260] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.342918 3601 ts_tablet_manager.cc:1398] T 5db25d87e72347f4a4410bcde440f359 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:41.343181 3602 ts_tablet_manager.cc:1398] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:41.343294 3600 raft_consensus.cc:348] T 90f51696d59c4f11bd44050fde07c276 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:41.343398 3600 raft_consensus.cc:374] T 90f51696d59c4f11bd44050fde07c276 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.343438 3602 tablet_bootstrap.cc:492] T 36bf7d2992034055bccfd8146449b696 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:41.343443 3600 raft_consensus.cc:729] T 90f51696d59c4f11bd44050fde07c276 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.343550 3600 consensus_queue.cc:260] T 90f51696d59c4f11bd44050fde07c276 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:41.344319 3602 tablet_bootstrap.cc:654] T 36bf7d2992034055bccfd8146449b696 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.344529 3606 raft_consensus.cc:348] T 90f51696d59c4f11bd44050fde07c276 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:41.344640 3606 raft_consensus.cc:374] T 90f51696d59c4f11bd44050fde07c276 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.344673 3606 raft_consensus.cc:729] T 90f51696d59c4f11bd44050fde07c276 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.344780 3606 consensus_queue.cc:260] T 90f51696d59c4f11bd44050fde07c276 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:41.345103 3606 ts_tablet_manager.cc:1398] T 90f51696d59c4f11bd44050fde07c276 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:41.346386 3602 tablet_bootstrap.cc:492] T 36bf7d2992034055bccfd8146449b696 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:41.346467 3602 ts_tablet_manager.cc:1367] T 36bf7d2992034055bccfd8146449b696 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:41.346915 3602 raft_consensus.cc:348] T 36bf7d2992034055bccfd8146449b696 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.347009 3602 raft_consensus.cc:374] T 36bf7d2992034055bccfd8146449b696 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.347030 3602 raft_consensus.cc:729] T 36bf7d2992034055bccfd8146449b696 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.347127 3602 consensus_queue.cc:260] T 36bf7d2992034055bccfd8146449b696 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.347352 3602 ts_tablet_manager.cc:1398] T 36bf7d2992034055bccfd8146449b696 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:41.347523 3602 tablet_bootstrap.cc:492] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:41.347580 3600 ts_tablet_manager.cc:1398] T 90f51696d59c4f11bd44050fde07c276 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:41.348364 3602 tablet_bootstrap.cc:654] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.349974 3602 tablet_bootstrap.cc:492] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:41.350306 3602 ts_tablet_manager.cc:1367] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:41.350800 3602 raft_consensus.cc:348] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:41.350981 3602 raft_consensus.cc:374] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.351045 3602 raft_consensus.cc:729] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.351166 3602 consensus_queue.cc:260] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:41.351609 3602 ts_tablet_manager.cc:1398] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:41.363495 30264 consensus_queue.cc:237] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.364748 30406 raft_consensus.cc:1261] T 7e1895c0656d49c09266cdbd231497cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:41.364984 30013 raft_consensus.cc:1261] T 7e1895c0656d49c09266cdbd231497cb P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:41.364996 30212 consensus_peers.cc:485] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): Couldn't send request to peer 9ecd907cf5634ac98f3f54f3dbec6849. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e1895c0656d49c09266cdbd231497cb. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:41.365168 2956 consensus_queue.cc:1035] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:41.365684 2956 consensus_queue.cc:1035] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:41.367676 3062 raft_consensus.cc:2932] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:41.368093 30013 raft_consensus.cc:2932] T 7e1895c0656d49c09266cdbd231497cb P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:41.369325 30406 raft_consensus.cc:2932] T 7e1895c0656d49c09266cdbd231497cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:41.370136 29866 catalog_manager.cc:5484] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:41.389930 32685 consensus_queue.cc:237] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:41.391273 348 raft_consensus.cc:1261] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:41.391301 32415 raft_consensus.cc:1261] T 0ca8c2fa60c543dab6f30acce1958573 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:41.391398 32620 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 -> Peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): Couldn't send request to peer 1d0bb69611af4876a2b3b33c4766e13a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ca8c2fa60c543dab6f30acce1958573. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:41.391687 3648 consensus_queue.cc:1035] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:41.391862 3042 consensus_queue.cc:1035] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:41.393793 3042 raft_consensus.cc:2932] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:41.394433 348 raft_consensus.cc:2932] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:41.395483 3376 raft_consensus.cc:482] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:41.395570 3376 raft_consensus.cc:504] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.395931 3376 leader_election.cc:288] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:41.396303 29872 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:41.396440 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5db25d87e72347f4a4410bcde440f359" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:41.396531 32685 raft_consensus.cc:2454] T 5db25d87e72347f4a4410bcde440f359 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:41.396888 29870 catalog_manager.cc:5484] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 reported cstate change: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:41.397500 32416 raft_consensus.cc:2932] T 0ca8c2fa60c543dab6f30acce1958573 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:41.399122 32750 leader_election.cc:302] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:41.399291 3376 raft_consensus.cc:2781] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:41.399355 3376 raft_consensus.cc:482] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:41.399395 3376 raft_consensus.cc:3037] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.400483 3376 raft_consensus.cc:504] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.400841 3376 leader_election.cc:288] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:41.401896 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5db25d87e72347f4a4410bcde440f359" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:41.402009 32685 raft_consensus.cc:3037] T 5db25d87e72347f4a4410bcde440f359 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.402930 32685 raft_consensus.cc:2454] T 5db25d87e72347f4a4410bcde440f359 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:41.403206 32750 leader_election.cc:302] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:41.403360 3376 raft_consensus.cc:2781] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:41.403421 3376 raft_consensus.cc:686] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:41.403508 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5db25d87e72347f4a4410bcde440f359" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:41.403530 3376 consensus_queue.cc:237] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.403606 32416 raft_consensus.cc:3037] T 5db25d87e72347f4a4410bcde440f359 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.403632 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5db25d87e72347f4a4410bcde440f359" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:41.404476 32416 raft_consensus.cc:2454] T 5db25d87e72347f4a4410bcde440f359 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:41.407328 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 331e53e2abef4e43a9fbe4e6f4b67229 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8d598c3820f7442b87a92f24c4f44125]), partition=
I20240501 14:09:41.407500 327 tablet_service.cc:1467] Processing CreateTablet for tablet 331e53e2abef4e43a9fbe4e6f4b67229 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8d598c3820f7442b87a92f24c4f44125]), partition=
I20240501 14:09:41.407586 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331e53e2abef4e43a9fbe4e6f4b67229. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.407706 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331e53e2abef4e43a9fbe4e6f4b67229. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.407792 29871 catalog_manager.cc:5484] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:41.409768 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 331e53e2abef4e43a9fbe4e6f4b67229 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8d598c3820f7442b87a92f24c4f44125]), partition=
I20240501 14:09:41.410005 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331e53e2abef4e43a9fbe4e6f4b67229. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.410279 3658 tablet_bootstrap.cc:492] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:41.411437 3658 tablet_bootstrap.cc:654] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.412093 3601 tablet_bootstrap.cc:492] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:41.412976 3601 tablet_bootstrap.cc:654] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.413560 3602 tablet_bootstrap.cc:492] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:41.414451 3602 tablet_bootstrap.cc:654] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.415414 3601 tablet_bootstrap.cc:492] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:41.415468 3602 tablet_bootstrap.cc:492] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:41.415491 3601 ts_tablet_manager.cc:1367] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:41.415546 3602 ts_tablet_manager.cc:1367] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:41.415899 3601 raft_consensus.cc:348] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:41.415910 3602 raft_consensus.cc:348] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:41.415993 3602 raft_consensus.cc:374] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.415987 3601 raft_consensus.cc:374] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.416014 3602 raft_consensus.cc:729] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.416018 3601 raft_consensus.cc:729] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.416124 3602 consensus_queue.cc:260] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:41.416123 3601 consensus_queue.cc:260] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:41.416394 3602 ts_tablet_manager.cc:1398] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:41.416747 3601 ts_tablet_manager.cc:1398] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:41.417879 3658 tablet_bootstrap.cc:492] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:41.418030 3658 ts_tablet_manager.cc:1367] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:41.418759 3658 raft_consensus.cc:348] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:41.418923 3658 raft_consensus.cc:374] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.419018 3658 raft_consensus.cc:729] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.419194 3658 consensus_queue.cc:260] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:41.419567 3658 ts_tablet_manager.cc:1398] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:41.437794 32416 consensus_queue.cc:237] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
W20240501 14:09:41.439354 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Couldn't send request to peer d35c16dcb506426993811bb2519d7db3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 575e8cde863540b09a52fe590cbf2bfc. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:41.439437 32553 raft_consensus.cc:1261] T 575e8cde863540b09a52fe590cbf2bfc P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:41.439818 2944 consensus_queue.cc:1035] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:41.440593 32685 raft_consensus.cc:1261] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:41.440953 2944 consensus_queue.cc:1035] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:41.442724 2944 raft_consensus.cc:2932] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:41.443138 32685 raft_consensus.cc:2932] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:41.446177 29871 catalog_manager.cc:5484] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38 reported cstate change: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:41.449237 32553 raft_consensus.cc:2932] T 575e8cde863540b09a52fe590cbf2bfc P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:41.493589 3126 raft_consensus.cc:482] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:41.493739 3126 raft_consensus.cc:504] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:41.494143 3126 leader_election.cc:288] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:41.494621 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f51696d59c4f11bd44050fde07c276" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:41.494773 30013 raft_consensus.cc:2454] T 90f51696d59c4f11bd44050fde07c276 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:41.494915 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f51696d59c4f11bd44050fde07c276" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:41.495064 32416 raft_consensus.cc:2454] T 90f51696d59c4f11bd44050fde07c276 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:41.495123 32752 leader_election.cc:302] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:41.495373 3126 raft_consensus.cc:2781] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:41.495433 3126 raft_consensus.cc:482] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:41.495482 3126 raft_consensus.cc:3037] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.496636 3126 raft_consensus.cc:504] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:41.496973 3126 leader_election.cc:288] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:41.497830 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f51696d59c4f11bd44050fde07c276" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:41.497934 30013 raft_consensus.cc:3037] T 90f51696d59c4f11bd44050fde07c276 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.498083 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f51696d59c4f11bd44050fde07c276" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:41.498184 32416 raft_consensus.cc:3037] T 90f51696d59c4f11bd44050fde07c276 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.498986 30013 raft_consensus.cc:2454] T 90f51696d59c4f11bd44050fde07c276 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:41.499233 32416 raft_consensus.cc:2454] T 90f51696d59c4f11bd44050fde07c276 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:41.499329 32752 leader_election.cc:302] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:41.499596 3126 raft_consensus.cc:2781] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:41.499691 3126 raft_consensus.cc:686] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:41.499832 3126 consensus_queue.cc:237] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:41.501835 29871 catalog_manager.cc:5484] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:41.629575 3557 raft_consensus.cc:482] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:41.629736 3557 raft_consensus.cc:504] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.630092 3557 leader_election.cc:288] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:41.630438 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bf7d2992034055bccfd8146449b696" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:41.630510 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bf7d2992034055bccfd8146449b696" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:41.630591 348 raft_consensus.cc:2454] T 36bf7d2992034055bccfd8146449b696 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:41.630623 32685 raft_consensus.cc:2454] T 36bf7d2992034055bccfd8146449b696 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:41.630853 30210 leader_election.cc:302] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:41.631089 3062 raft_consensus.cc:2781] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:41.631155 3062 raft_consensus.cc:482] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:41.631211 3062 raft_consensus.cc:3037] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.632123 3062 raft_consensus.cc:504] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.632458 3062 leader_election.cc:288] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:41.632829 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bf7d2992034055bccfd8146449b696" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:41.632807 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bf7d2992034055bccfd8146449b696" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:41.632954 348 raft_consensus.cc:3037] T 36bf7d2992034055bccfd8146449b696 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.633000 32685 raft_consensus.cc:3037] T 36bf7d2992034055bccfd8146449b696 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.633952 32685 raft_consensus.cc:2454] T 36bf7d2992034055bccfd8146449b696 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:41.633944 348 raft_consensus.cc:2454] T 36bf7d2992034055bccfd8146449b696 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:41.634223 30210 leader_election.cc:302] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:41.634399 3062 raft_consensus.cc:2781] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:41.634495 3062 raft_consensus.cc:686] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:41.634618 3062 consensus_queue.cc:237] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.636293 29864 catalog_manager.cc:5484] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:41.663882 3719 ts_tablet_manager.cc:902] T 588dd26e1820409a9dfccd8ea6a30f49 P 1d0bb69611af4876a2b3b33c4766e13a: Initiating tablet copy from peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:41.664592 3719 tablet_copy_client.cc:308] T 588dd26e1820409a9dfccd8ea6a30f49 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.133:46265
I20240501 14:09:41.665096 32443 tablet_copy_service.cc:140] P a91029b90bf9415a89a2f70736759ad3: Received BeginTabletCopySession request for tablet 588dd26e1820409a9dfccd8ea6a30f49 from peer 1d0bb69611af4876a2b3b33c4766e13a ({username='slave'} at 127.22.252.131:41881)
I20240501 14:09:41.665191 32443 tablet_copy_service.cc:161] P a91029b90bf9415a89a2f70736759ad3: Beginning new tablet copy session on tablet 588dd26e1820409a9dfccd8ea6a30f49 from peer 1d0bb69611af4876a2b3b33c4766e13a at {username='slave'} at 127.22.252.131:41881: session id = 1d0bb69611af4876a2b3b33c4766e13a-588dd26e1820409a9dfccd8ea6a30f49
I20240501 14:09:41.665885 32443 tablet_copy_source_session.cc:205] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:41.667001 3719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 588dd26e1820409a9dfccd8ea6a30f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.669173 3719 tablet_copy_client.cc:783] T 588dd26e1820409a9dfccd8ea6a30f49 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:41.669356 3719 tablet_copy_client.cc:647] T 588dd26e1820409a9dfccd8ea6a30f49 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:41.670645 3719 tablet_copy_client.cc:523] T 588dd26e1820409a9dfccd8ea6a30f49 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:41.671860 3719 tablet_bootstrap.cc:492] T 588dd26e1820409a9dfccd8ea6a30f49 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:41.673242 3724 ts_tablet_manager.cc:902] T 4a5869fb581b4939ad2207ab9200d674 P a91029b90bf9415a89a2f70736759ad3: Initiating tablet copy from peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:41.673758 3724 tablet_copy_client.cc:308] T 4a5869fb581b4939ad2207ab9200d674 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.131:43103
I20240501 14:09:41.674149 30297 tablet_copy_service.cc:140] P 1d0bb69611af4876a2b3b33c4766e13a: Received BeginTabletCopySession request for tablet 4a5869fb581b4939ad2207ab9200d674 from peer a91029b90bf9415a89a2f70736759ad3 ({username='slave'} at 127.22.252.133:60853)
I20240501 14:09:41.674252 30297 tablet_copy_service.cc:161] P 1d0bb69611af4876a2b3b33c4766e13a: Beginning new tablet copy session on tablet 4a5869fb581b4939ad2207ab9200d674 from peer a91029b90bf9415a89a2f70736759ad3 at {username='slave'} at 127.22.252.133:60853: session id = a91029b90bf9415a89a2f70736759ad3-4a5869fb581b4939ad2207ab9200d674
I20240501 14:09:41.674978 30297 tablet_copy_source_session.cc:205] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:41.675732 3724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a5869fb581b4939ad2207ab9200d674. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.675987 3719 tablet_bootstrap.cc:492] T 588dd26e1820409a9dfccd8ea6a30f49 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:41.676290 3719 tablet_bootstrap.cc:492] T 588dd26e1820409a9dfccd8ea6a30f49 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap complete.
I20240501 14:09:41.676445 3719 ts_tablet_manager.cc:1367] T 588dd26e1820409a9dfccd8ea6a30f49 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240501 14:09:41.676867 3719 raft_consensus.cc:348] T 588dd26e1820409a9dfccd8ea6a30f49 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:41.676959 3719 raft_consensus.cc:729] T 588dd26e1820409a9dfccd8ea6a30f49 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.677074 3719 consensus_queue.cc:260] T 588dd26e1820409a9dfccd8ea6a30f49 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:41.677817 3719 ts_tablet_manager.cc:1398] T 588dd26e1820409a9dfccd8ea6a30f49 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:41.677912 3724 tablet_copy_client.cc:783] T 4a5869fb581b4939ad2207ab9200d674 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:41.678062 3724 tablet_copy_client.cc:647] T 4a5869fb581b4939ad2207ab9200d674 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:41.679004 32443 tablet_copy_service.cc:342] P a91029b90bf9415a89a2f70736759ad3: Request end of tablet copy session 1d0bb69611af4876a2b3b33c4766e13a-588dd26e1820409a9dfccd8ea6a30f49 received from {username='slave'} at 127.22.252.131:41881
I20240501 14:09:41.679078 32443 tablet_copy_service.cc:433] P a91029b90bf9415a89a2f70736759ad3: ending tablet copy session 1d0bb69611af4876a2b3b33c4766e13a-588dd26e1820409a9dfccd8ea6a30f49 on tablet 588dd26e1820409a9dfccd8ea6a30f49 with peer 1d0bb69611af4876a2b3b33c4766e13a
I20240501 14:09:41.679246 3724 tablet_copy_client.cc:523] T 4a5869fb581b4939ad2207ab9200d674 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:41.680562 3724 tablet_bootstrap.cc:492] T 4a5869fb581b4939ad2207ab9200d674 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:41.699549 3724 tablet_bootstrap.cc:492] T 4a5869fb581b4939ad2207ab9200d674 P a91029b90bf9415a89a2f70736759ad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:41.699566 3376 raft_consensus.cc:482] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:41.699678 3376 raft_consensus.cc:504] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:41.699857 3724 tablet_bootstrap.cc:492] T 4a5869fb581b4939ad2207ab9200d674 P a91029b90bf9415a89a2f70736759ad3: Bootstrap complete.
I20240501 14:09:41.700075 3724 ts_tablet_manager.cc:1367] T 4a5869fb581b4939ad2207ab9200d674 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20240501 14:09:41.700054 3376 leader_election.cc:288] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:41.700474 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331e53e2abef4e43a9fbe4e6f4b67229" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:41.700521 3724 raft_consensus.cc:348] T 4a5869fb581b4939ad2207ab9200d674 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:41.700619 3724 raft_consensus.cc:729] T 4a5869fb581b4939ad2207ab9200d674 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.700628 32553 raft_consensus.cc:2454] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:41.700738 3724 consensus_queue.cc:260] T 4a5869fb581b4939ad2207ab9200d674 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:41.700984 32753 leader_election.cc:302] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:41.701231 3126 raft_consensus.cc:2781] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:41.701289 3126 raft_consensus.cc:482] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:41.701335 3126 raft_consensus.cc:3037] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.701401 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331e53e2abef4e43a9fbe4e6f4b67229" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:41.701539 32685 raft_consensus.cc:2454] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:41.702271 3724 ts_tablet_manager.cc:1398] T 4a5869fb581b4939ad2207ab9200d674 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:41.702379 3126 raft_consensus.cc:504] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:41.702723 3126 leader_election.cc:288] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:41.703169 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331e53e2abef4e43a9fbe4e6f4b67229" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:41.703325 32553 raft_consensus.cc:3037] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.704505 32553 raft_consensus.cc:2454] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:41.704718 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331e53e2abef4e43a9fbe4e6f4b67229" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:41.704834 32685 raft_consensus.cc:3037] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.705689 32685 raft_consensus.cc:2454] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:41.705899 32753 leader_election.cc:302] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:41.706193 30297 tablet_copy_service.cc:342] P 1d0bb69611af4876a2b3b33c4766e13a: Request end of tablet copy session a91029b90bf9415a89a2f70736759ad3-4a5869fb581b4939ad2207ab9200d674 received from {username='slave'} at 127.22.252.133:60853
I20240501 14:09:41.706290 30297 tablet_copy_service.cc:433] P 1d0bb69611af4876a2b3b33c4766e13a: ending tablet copy session a91029b90bf9415a89a2f70736759ad3-4a5869fb581b4939ad2207ab9200d674 on tablet 4a5869fb581b4939ad2207ab9200d674 with peer a91029b90bf9415a89a2f70736759ad3
I20240501 14:09:41.706562 3126 raft_consensus.cc:2781] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:41.706657 3126 raft_consensus.cc:686] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:41.706782 3126 consensus_queue.cc:237] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:41.708592 29871 catalog_manager.cc:5484] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:41.733187 3748 ts_tablet_manager.cc:902] T 776d6722a14345f490f69f90d80d6a6a P d35c16dcb506426993811bb2519d7db3: Initiating tablet copy from peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:41.738056 3748 tablet_copy_client.cc:308] T 776d6722a14345f490f69f90d80d6a6a P d35c16dcb506426993811bb2519d7db3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.133:46265
I20240501 14:09:41.738685 32534 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:41.739934 32443 tablet_copy_service.cc:140] P a91029b90bf9415a89a2f70736759ad3: Received BeginTabletCopySession request for tablet 776d6722a14345f490f69f90d80d6a6a from peer d35c16dcb506426993811bb2519d7db3 ({username='slave'} at 127.22.252.130:47505)
I20240501 14:09:41.740026 32443 tablet_copy_service.cc:161] P a91029b90bf9415a89a2f70736759ad3: Beginning new tablet copy session on tablet 776d6722a14345f490f69f90d80d6a6a from peer d35c16dcb506426993811bb2519d7db3 at {username='slave'} at 127.22.252.130:47505: session id = d35c16dcb506426993811bb2519d7db3-776d6722a14345f490f69f90d80d6a6a
I20240501 14:09:41.741046 32443 tablet_copy_source_session.cc:205] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:41.743196 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:41.745699 30388 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:41.752274 32665 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:41.753327 327 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:41.754734 30124 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:41.757198 29993 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:41.757396 3748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776d6722a14345f490f69f90d80d6a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.760017 3748 tablet_copy_client.cc:783] T 776d6722a14345f490f69f90d80d6a6a P d35c16dcb506426993811bb2519d7db3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:41.760196 3748 tablet_copy_client.cc:647] T 776d6722a14345f490f69f90d80d6a6a P d35c16dcb506426993811bb2519d7db3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:41.765223 32400 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:41.772368 3748 tablet_copy_client.cc:523] T 776d6722a14345f490f69f90d80d6a6a P d35c16dcb506426993811bb2519d7db3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:41.774106 3748 tablet_bootstrap.cc:492] T 776d6722a14345f490f69f90d80d6a6a P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:41.781891 3753 ts_tablet_manager.cc:902] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38: Initiating tablet copy from peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:41.782421 3753 tablet_copy_client.cc:308] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.130:36457
I20240501 14:09:41.782840 30165 tablet_copy_service.cc:140] P d35c16dcb506426993811bb2519d7db3: Received BeginTabletCopySession request for tablet c362bd01a59b4bcf85eeb59713e58b64 from peer db935501e9104de5b15c8780681d6c38 ({username='slave'} at 127.22.252.135:49787)
I20240501 14:09:41.782931 30165 tablet_copy_service.cc:161] P d35c16dcb506426993811bb2519d7db3: Beginning new tablet copy session on tablet c362bd01a59b4bcf85eeb59713e58b64 from peer db935501e9104de5b15c8780681d6c38 at {username='slave'} at 127.22.252.135:49787: session id = db935501e9104de5b15c8780681d6c38-c362bd01a59b4bcf85eeb59713e58b64
I20240501 14:09:41.783802 30165 tablet_copy_source_session.cc:205] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:41.785341 3753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c362bd01a59b4bcf85eeb59713e58b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.787778 3753 tablet_copy_client.cc:783] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:41.787918 3753 tablet_copy_client.cc:647] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:41.789078 3753 tablet_copy_client.cc:523] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:41.796161 3753 tablet_bootstrap.cc:492] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:41.803669 3748 tablet_bootstrap.cc:492] T 776d6722a14345f490f69f90d80d6a6a P d35c16dcb506426993811bb2519d7db3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:41.803952 3748 tablet_bootstrap.cc:492] T 776d6722a14345f490f69f90d80d6a6a P d35c16dcb506426993811bb2519d7db3: Bootstrap complete.
I20240501 14:09:41.804093 3748 ts_tablet_manager.cc:1367] T 776d6722a14345f490f69f90d80d6a6a P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.000s
I20240501 14:09:41.804550 3748 raft_consensus.cc:348] T 776d6722a14345f490f69f90d80d6a6a P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:41.804656 3748 raft_consensus.cc:729] T 776d6722a14345f490f69f90d80d6a6a P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.804772 3748 consensus_queue.cc:260] T 776d6722a14345f490f69f90d80d6a6a P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:41.805032 3748 ts_tablet_manager.cc:1398] T 776d6722a14345f490f69f90d80d6a6a P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:41.806581 32443 tablet_copy_service.cc:342] P a91029b90bf9415a89a2f70736759ad3: Request end of tablet copy session d35c16dcb506426993811bb2519d7db3-776d6722a14345f490f69f90d80d6a6a received from {username='slave'} at 127.22.252.130:47505
I20240501 14:09:41.806653 32443 tablet_copy_service.cc:433] P a91029b90bf9415a89a2f70736759ad3: ending tablet copy session d35c16dcb506426993811bb2519d7db3-776d6722a14345f490f69f90d80d6a6a on tablet 776d6722a14345f490f69f90d80d6a6a with peer d35c16dcb506426993811bb2519d7db3
I20240501 14:09:41.823346 3376 consensus_queue.cc:1035] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:41.826455 3126 consensus_queue.cc:1035] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:41.831123 3753 tablet_bootstrap.cc:492] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:41.831815 3753 tablet_bootstrap.cc:492] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38: Bootstrap complete.
I20240501 14:09:41.832034 3753 ts_tablet_manager.cc:1367] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.036s user 0.027s sys 0.000s
I20240501 14:09:41.832532 3753 raft_consensus.cc:348] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:41.832675 3753 raft_consensus.cc:729] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.832796 3753 consensus_queue.cc:260] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:41.833076 3753 ts_tablet_manager.cc:1398] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:41.833897 30165 tablet_copy_service.cc:342] P d35c16dcb506426993811bb2519d7db3: Request end of tablet copy session db935501e9104de5b15c8780681d6c38-c362bd01a59b4bcf85eeb59713e58b64 received from {username='slave'} at 127.22.252.135:49787
I20240501 14:09:41.833987 30165 tablet_copy_service.cc:433] P d35c16dcb506426993811bb2519d7db3: ending tablet copy session db935501e9104de5b15c8780681d6c38-c362bd01a59b4bcf85eeb59713e58b64 on tablet c362bd01a59b4bcf85eeb59713e58b64 with peer db935501e9104de5b15c8780681d6c38
I20240501 14:09:41.846815 3777 ts_tablet_manager.cc:902] T ee998c3fcf3449d5845ac1c950f326d7 P 9ecd907cf5634ac98f3f54f3dbec6849: Initiating tablet copy from peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:41.847270 3777 tablet_copy_client.cc:308] T ee998c3fcf3449d5845ac1c950f326d7 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.130:36457
I20240501 14:09:41.847703 30165 tablet_copy_service.cc:140] P d35c16dcb506426993811bb2519d7db3: Received BeginTabletCopySession request for tablet ee998c3fcf3449d5845ac1c950f326d7 from peer 9ecd907cf5634ac98f3f54f3dbec6849 ({username='slave'} at 127.22.252.136:49091)
I20240501 14:09:41.847801 30165 tablet_copy_service.cc:161] P d35c16dcb506426993811bb2519d7db3: Beginning new tablet copy session on tablet ee998c3fcf3449d5845ac1c950f326d7 from peer 9ecd907cf5634ac98f3f54f3dbec6849 at {username='slave'} at 127.22.252.136:49091: session id = 9ecd907cf5634ac98f3f54f3dbec6849-ee998c3fcf3449d5845ac1c950f326d7
I20240501 14:09:41.848799 30165 tablet_copy_source_session.cc:205] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:41.849800 3777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee998c3fcf3449d5845ac1c950f326d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.851660 3777 tablet_copy_client.cc:783] T ee998c3fcf3449d5845ac1c950f326d7 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:41.851820 3777 tablet_copy_client.cc:647] T ee998c3fcf3449d5845ac1c950f326d7 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:41.853101 3777 tablet_copy_client.cc:523] T ee998c3fcf3449d5845ac1c950f326d7 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:41.854002 3777 tablet_bootstrap.cc:492] T ee998c3fcf3449d5845ac1c950f326d7 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:41.877156 3777 tablet_bootstrap.cc:492] T ee998c3fcf3449d5845ac1c950f326d7 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:41.877492 3777 tablet_bootstrap.cc:492] T ee998c3fcf3449d5845ac1c950f326d7 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap complete.
I20240501 14:09:41.877875 3777 ts_tablet_manager.cc:1367] T ee998c3fcf3449d5845ac1c950f326d7 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.005s
I20240501 14:09:41.878329 3777 raft_consensus.cc:348] T ee998c3fcf3449d5845ac1c950f326d7 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.878463 3777 raft_consensus.cc:729] T ee998c3fcf3449d5845ac1c950f326d7 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.878583 3777 consensus_queue.cc:260] T ee998c3fcf3449d5845ac1c950f326d7 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:41.878890 3777 ts_tablet_manager.cc:1398] T ee998c3fcf3449d5845ac1c950f326d7 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:41.879596 30165 tablet_copy_service.cc:342] P d35c16dcb506426993811bb2519d7db3: Request end of tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-ee998c3fcf3449d5845ac1c950f326d7 received from {username='slave'} at 127.22.252.136:49091
I20240501 14:09:41.879678 30165 tablet_copy_service.cc:433] P d35c16dcb506426993811bb2519d7db3: ending tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-ee998c3fcf3449d5845ac1c950f326d7 on tablet ee998c3fcf3449d5845ac1c950f326d7 with peer 9ecd907cf5634ac98f3f54f3dbec6849
I20240501 14:09:41.883235 29873 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:41.887506 32685 raft_consensus.cc:1261] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:41.887743 32553 raft_consensus.cc:1261] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:41.887877 3376 consensus_queue.cc:1035] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:41.888164 3376 consensus_queue.cc:1035] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:41.891052 29873 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet 331e53e2abef4e43a9fbe4e6f4b67229 (table rebalancer_dynamic_table [id=8d598c3820f7442b87a92f24c4f44125]). Expected version 1 got 0
I20240501 14:09:41.892213 3126 consensus_queue.cc:1035] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:41.892349 29864 catalog_manager.cc:5576] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497) does not have the latest schema for tablet 331e53e2abef4e43a9fbe4e6f4b67229 (table rebalancer_dynamic_table [id=8d598c3820f7442b87a92f24c4f44125]). Expected version 1 got 0
I20240501 14:09:41.895968 3821 tablet.cc:1722] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:41.896384 3821 tablet.cc:1620] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:41.898048 29864 catalog_manager.cc:5576] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497) does not have the latest schema for tablet 331e53e2abef4e43a9fbe4e6f4b67229 (table rebalancer_dynamic_table [id=8d598c3820f7442b87a92f24c4f44125]). Expected version 1 got 0
I20240501 14:09:41.898072 3822 tablet.cc:1722] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:41.898201 3376 consensus_queue.cc:1035] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:41.898424 3822 tablet.cc:1620] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:41.898954 3820 tablet.cc:1722] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:41.899220 3820 tablet.cc:1620] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:41.904996 29873 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=8d598c3820f7442b87a92f24c4f44125] alter complete (version 1)
I20240501 14:09:41.920536 3748 ts_tablet_manager.cc:902] T 575e8cde863540b09a52fe590cbf2bfc P d35c16dcb506426993811bb2519d7db3: Initiating tablet copy from peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:41.921017 3748 tablet_copy_client.cc:308] T 575e8cde863540b09a52fe590cbf2bfc P d35c16dcb506426993811bb2519d7db3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.133:46265
I20240501 14:09:41.921500 32443 tablet_copy_service.cc:140] P a91029b90bf9415a89a2f70736759ad3: Received BeginTabletCopySession request for tablet 575e8cde863540b09a52fe590cbf2bfc from peer d35c16dcb506426993811bb2519d7db3 ({username='slave'} at 127.22.252.130:47505)
I20240501 14:09:41.921607 32443 tablet_copy_service.cc:161] P a91029b90bf9415a89a2f70736759ad3: Beginning new tablet copy session on tablet 575e8cde863540b09a52fe590cbf2bfc from peer d35c16dcb506426993811bb2519d7db3 at {username='slave'} at 127.22.252.130:47505: session id = d35c16dcb506426993811bb2519d7db3-575e8cde863540b09a52fe590cbf2bfc
I20240501 14:09:41.922595 32443 tablet_copy_source_session.cc:205] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:41.923604 3748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 575e8cde863540b09a52fe590cbf2bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.925755 3748 tablet_copy_client.cc:783] T 575e8cde863540b09a52fe590cbf2bfc P d35c16dcb506426993811bb2519d7db3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:41.925907 3748 tablet_copy_client.cc:647] T 575e8cde863540b09a52fe590cbf2bfc P d35c16dcb506426993811bb2519d7db3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:41.926950 3748 tablet_copy_client.cc:523] T 575e8cde863540b09a52fe590cbf2bfc P d35c16dcb506426993811bb2519d7db3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:41.927826 3748 tablet_bootstrap.cc:492] T 575e8cde863540b09a52fe590cbf2bfc P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:41.930792 3748 tablet_bootstrap.cc:492] T 575e8cde863540b09a52fe590cbf2bfc P d35c16dcb506426993811bb2519d7db3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:41.931107 3748 tablet_bootstrap.cc:492] T 575e8cde863540b09a52fe590cbf2bfc P d35c16dcb506426993811bb2519d7db3: Bootstrap complete.
I20240501 14:09:41.931286 3748 ts_tablet_manager.cc:1367] T 575e8cde863540b09a52fe590cbf2bfc P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.003s user 0.004s sys 0.000s
I20240501 14:09:41.931803 3748 raft_consensus.cc:348] T 575e8cde863540b09a52fe590cbf2bfc P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:41.931907 3748 raft_consensus.cc:729] T 575e8cde863540b09a52fe590cbf2bfc P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.932221 3748 consensus_queue.cc:260] T 575e8cde863540b09a52fe590cbf2bfc P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:41.933116 3748 ts_tablet_manager.cc:1398] T 575e8cde863540b09a52fe590cbf2bfc P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:41.933938 32443 tablet_copy_service.cc:342] P a91029b90bf9415a89a2f70736759ad3: Request end of tablet copy session d35c16dcb506426993811bb2519d7db3-575e8cde863540b09a52fe590cbf2bfc received from {username='slave'} at 127.22.252.130:47505
I20240501 14:09:41.934018 32443 tablet_copy_service.cc:433] P a91029b90bf9415a89a2f70736759ad3: ending tablet copy session d35c16dcb506426993811bb2519d7db3-575e8cde863540b09a52fe590cbf2bfc on tablet 575e8cde863540b09a52fe590cbf2bfc with peer d35c16dcb506426993811bb2519d7db3
I20240501 14:09:41.943536 32534 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:41.944350 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:41.950925 327 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:41.951826 3777 ts_tablet_manager.cc:902] T 7e1895c0656d49c09266cdbd231497cb P 9ecd907cf5634ac98f3f54f3dbec6849: Initiating tablet copy from peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:41.952318 3777 tablet_copy_client.cc:308] T 7e1895c0656d49c09266cdbd231497cb P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.131:43103
I20240501 14:09:41.952816 30297 tablet_copy_service.cc:140] P 1d0bb69611af4876a2b3b33c4766e13a: Received BeginTabletCopySession request for tablet 7e1895c0656d49c09266cdbd231497cb from peer 9ecd907cf5634ac98f3f54f3dbec6849 ({username='slave'} at 127.22.252.136:55245)
I20240501 14:09:41.952904 30297 tablet_copy_service.cc:161] P 1d0bb69611af4876a2b3b33c4766e13a: Beginning new tablet copy session on tablet 7e1895c0656d49c09266cdbd231497cb from peer 9ecd907cf5634ac98f3f54f3dbec6849 at {username='slave'} at 127.22.252.136:55245: session id = 9ecd907cf5634ac98f3f54f3dbec6849-7e1895c0656d49c09266cdbd231497cb
I20240501 14:09:41.953491 30388 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:41.953876 30297 tablet_copy_source_session.cc:205] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:41.954448 32400 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:41.954782 3777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e1895c0656d49c09266cdbd231497cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.956820 3777 tablet_copy_client.cc:783] T 7e1895c0656d49c09266cdbd231497cb P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:41.956946 3777 tablet_copy_client.cc:647] T 7e1895c0656d49c09266cdbd231497cb P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:41.960512 3777 tablet_copy_client.cc:523] T 7e1895c0656d49c09266cdbd231497cb P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:41.961622 3777 tablet_bootstrap.cc:492] T 7e1895c0656d49c09266cdbd231497cb P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:41.972369 30124 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:41.979146 29993 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:42.001878 32665 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:42.007403 29866 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:42.009968 3719 ts_tablet_manager.cc:902] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a: Initiating tablet copy from peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:42.010411 3719 tablet_copy_client.cc:308] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.135:41915
I20240501 14:09:42.010936 32705 tablet_copy_service.cc:140] P db935501e9104de5b15c8780681d6c38: Received BeginTabletCopySession request for tablet 0ca8c2fa60c543dab6f30acce1958573 from peer 1d0bb69611af4876a2b3b33c4766e13a ({username='slave'} at 127.22.252.131:49203)
I20240501 14:09:42.011077 32705 tablet_copy_service.cc:161] P db935501e9104de5b15c8780681d6c38: Beginning new tablet copy session on tablet 0ca8c2fa60c543dab6f30acce1958573 from peer 1d0bb69611af4876a2b3b33c4766e13a at {username='slave'} at 127.22.252.131:49203: session id = 1d0bb69611af4876a2b3b33c4766e13a-0ca8c2fa60c543dab6f30acce1958573
I20240501 14:09:42.012416 32705 tablet_copy_source_session.cc:205] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:42.013443 3719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca8c2fa60c543dab6f30acce1958573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.015532 3820 tablet.cc:1722] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:42.015606 3719 tablet_copy_client.cc:783] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:42.015746 3719 tablet_copy_client.cc:647] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:42.015803 3820 tablet.cc:1620] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:42.021257 3822 tablet.cc:1722] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:42.021557 3822 tablet.cc:1620] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:42.024859 29866 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=8d598c3820f7442b87a92f24c4f44125] alter complete (version 2)
I20240501 14:09:42.026559 3821 tablet.cc:1722] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:42.026824 3821 tablet.cc:1620] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:42.029418 3719 tablet_copy_client.cc:523] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:42.030416 3777 tablet_bootstrap.cc:492] T 7e1895c0656d49c09266cdbd231497cb P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:42.030628 3719 tablet_bootstrap.cc:492] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:42.030673 3777 tablet_bootstrap.cc:492] T 7e1895c0656d49c09266cdbd231497cb P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap complete.
I20240501 14:09:42.030803 3777 ts_tablet_manager.cc:1367] T 7e1895c0656d49c09266cdbd231497cb P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.069s user 0.024s sys 0.000s
I20240501 14:09:42.031199 3777 raft_consensus.cc:348] T 7e1895c0656d49c09266cdbd231497cb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:42.031296 3777 raft_consensus.cc:729] T 7e1895c0656d49c09266cdbd231497cb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.031440 3777 consensus_queue.cc:260] T 7e1895c0656d49c09266cdbd231497cb P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:42.031744 3777 ts_tablet_manager.cc:1398] T 7e1895c0656d49c09266cdbd231497cb P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:42.032186 30297 tablet_copy_service.cc:342] P 1d0bb69611af4876a2b3b33c4766e13a: Request end of tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-7e1895c0656d49c09266cdbd231497cb received from {username='slave'} at 127.22.252.136:55245
I20240501 14:09:42.032292 30297 tablet_copy_service.cc:433] P 1d0bb69611af4876a2b3b33c4766e13a: ending tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-7e1895c0656d49c09266cdbd231497cb on tablet 7e1895c0656d49c09266cdbd231497cb with peer 9ecd907cf5634ac98f3f54f3dbec6849
I20240501 14:09:42.035051 3719 tablet_bootstrap.cc:492] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:42.035286 3719 tablet_bootstrap.cc:492] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap complete.
I20240501 14:09:42.035434 3719 ts_tablet_manager.cc:1367] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 14:09:42.035892 3719 raft_consensus.cc:348] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:42.036047 3719 raft_consensus.cc:729] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.036190 3719 consensus_queue.cc:260] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:42.036562 3719 ts_tablet_manager.cc:1398] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:42.037047 32705 tablet_copy_service.cc:342] P db935501e9104de5b15c8780681d6c38: Request end of tablet copy session 1d0bb69611af4876a2b3b33c4766e13a-0ca8c2fa60c543dab6f30acce1958573 received from {username='slave'} at 127.22.252.131:49203
I20240501 14:09:42.037129 32705 tablet_copy_service.cc:433] P db935501e9104de5b15c8780681d6c38: ending tablet copy session 1d0bb69611af4876a2b3b33c4766e13a-0ca8c2fa60c543dab6f30acce1958573 on tablet 0ca8c2fa60c543dab6f30acce1958573 with peer 1d0bb69611af4876a2b3b33c4766e13a
I20240501 14:09:42.078099 32416 raft_consensus.cc:1203] T 4a5869fb581b4939ad2207ab9200d674 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:42.079110 30264 raft_consensus.cc:1203] T 588dd26e1820409a9dfccd8ea6a30f49 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:42.126752 29866 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:42.132963 3820 tablet.cc:1722] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:42.133544 3821 tablet.cc:1722] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:42.137396 3822 tablet.cc:1722] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:42.143260 29866 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=8d598c3820f7442b87a92f24c4f44125] alter complete (version 3)
I20240501 14:09:42.153414 32534 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:42.156286 30124 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:42.158740 32400 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:42.164083 29993 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:42.164899 32665 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:42.168279 327 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:42.168414 30388 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:42.191438 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:42.209230 29866 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:42.211329 3062 consensus_queue.cc:1035] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:42.214864 3062 consensus_queue.cc:1035] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:42.224000 32534 tablet_service.cc:1467] Processing CreateTablet for tablet a63e9bcd27234f2f9a9e0b268fd73486 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=8c1db6a412e9480587e107e98904d00e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.224352 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63e9bcd27234f2f9a9e0b268fd73486. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.224537 30256 tablet_service.cc:1467] Processing CreateTablet for tablet a63e9bcd27234f2f9a9e0b268fd73486 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=8c1db6a412e9480587e107e98904d00e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.224642 327 tablet_service.cc:1467] Processing CreateTablet for tablet a63e9bcd27234f2f9a9e0b268fd73486 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=8c1db6a412e9480587e107e98904d00e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.224774 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63e9bcd27234f2f9a9e0b268fd73486. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.224879 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63e9bcd27234f2f9a9e0b268fd73486. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.226984 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 6229c0c3e98c462b95f9bfe066367e87 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=8c1db6a412e9480587e107e98904d00e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.227222 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6229c0c3e98c462b95f9bfe066367e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.227279 3911 tablet_bootstrap.cc:492] T a63e9bcd27234f2f9a9e0b268fd73486 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:42.228425 3911 tablet_bootstrap.cc:654] T a63e9bcd27234f2f9a9e0b268fd73486 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.228387 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 6229c0c3e98c462b95f9bfe066367e87 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=8c1db6a412e9480587e107e98904d00e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.228523 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 22bb82c5d65640128fc3271212aafb60 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=8c1db6a412e9480587e107e98904d00e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.228603 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6229c0c3e98c462b95f9bfe066367e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.228749 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22bb82c5d65640128fc3271212aafb60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.229104 3912 tablet_bootstrap.cc:492] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:42.230226 3912 tablet_bootstrap.cc:654] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.230860 3914 tablet_bootstrap.cc:492] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:42.231030 3910 tablet_bootstrap.cc:492] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:42.231496 327 tablet_service.cc:1467] Processing CreateTablet for tablet 6229c0c3e98c462b95f9bfe066367e87 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=8c1db6a412e9480587e107e98904d00e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.231725 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6229c0c3e98c462b95f9bfe066367e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.231892 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 22bb82c5d65640128fc3271212aafb60 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=8c1db6a412e9480587e107e98904d00e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.232118 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22bb82c5d65640128fc3271212aafb60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.234160 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 22bb82c5d65640128fc3271212aafb60 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=8c1db6a412e9480587e107e98904d00e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.234423 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22bb82c5d65640128fc3271212aafb60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.234483 3914 tablet_bootstrap.cc:654] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.235186 3916 tablet_bootstrap.cc:492] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:42.236178 3916 tablet_bootstrap.cc:654] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.236810 3910 tablet_bootstrap.cc:654] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.237318 3912 tablet_bootstrap.cc:492] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:42.237411 3912 ts_tablet_manager.cc:1367] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:42.238148 3910 tablet_bootstrap.cc:492] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:42.238173 3912 raft_consensus.cc:348] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:42.238297 3912 raft_consensus.cc:374] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.238332 3912 raft_consensus.cc:729] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.238446 3912 consensus_queue.cc:260] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:42.238236 3910 ts_tablet_manager.cc:1367] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:42.238765 3912 ts_tablet_manager.cc:1398] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:42.238922 3912 tablet_bootstrap.cc:492] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:42.239090 3910 raft_consensus.cc:348] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:42.239181 3910 raft_consensus.cc:374] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.239221 3910 raft_consensus.cc:729] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.239377 3910 consensus_queue.cc:260] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:42.239804 3912 tablet_bootstrap.cc:654] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.240600 3910 ts_tablet_manager.cc:1398] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:42.241032 3914 tablet_bootstrap.cc:492] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:42.241175 3914 ts_tablet_manager.cc:1367] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:42.241192 3910 tablet_bootstrap.cc:492] T 6229c0c3e98c462b95f9bfe066367e87 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:42.241688 3914 raft_consensus.cc:348] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:42.242233 3911 tablet_bootstrap.cc:492] T a63e9bcd27234f2f9a9e0b268fd73486 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:42.242314 3911 ts_tablet_manager.cc:1367] T a63e9bcd27234f2f9a9e0b268fd73486 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 14:09:42.242974 3910 tablet_bootstrap.cc:654] T 6229c0c3e98c462b95f9bfe066367e87 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.243042 3911 raft_consensus.cc:348] T a63e9bcd27234f2f9a9e0b268fd73486 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:42.243150 3911 raft_consensus.cc:374] T a63e9bcd27234f2f9a9e0b268fd73486 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.243188 3911 raft_consensus.cc:729] T a63e9bcd27234f2f9a9e0b268fd73486 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.243306 3911 consensus_queue.cc:260] T a63e9bcd27234f2f9a9e0b268fd73486 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:42.243853 3911 ts_tablet_manager.cc:1398] T a63e9bcd27234f2f9a9e0b268fd73486 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:42.244637 3922 tablet_bootstrap.cc:492] T 22bb82c5d65640128fc3271212aafb60 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:42.245234 3910 tablet_bootstrap.cc:492] T 6229c0c3e98c462b95f9bfe066367e87 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:42.245280 3914 raft_consensus.cc:374] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.245316 3910 ts_tablet_manager.cc:1367] T 6229c0c3e98c462b95f9bfe066367e87 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:42.245325 3914 raft_consensus.cc:729] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.245458 3914 consensus_queue.cc:260] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:42.245733 3910 raft_consensus.cc:348] T 6229c0c3e98c462b95f9bfe066367e87 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:42.245826 3910 raft_consensus.cc:374] T 6229c0c3e98c462b95f9bfe066367e87 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.245855 3910 raft_consensus.cc:729] T 6229c0c3e98c462b95f9bfe066367e87 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.245945 3910 consensus_queue.cc:260] T 6229c0c3e98c462b95f9bfe066367e87 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:42.246557 3914 ts_tablet_manager.cc:1398] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:42.247078 3910 ts_tablet_manager.cc:1398] T 6229c0c3e98c462b95f9bfe066367e87 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:42.247291 3910 tablet_bootstrap.cc:492] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:42.247310 3912 tablet_bootstrap.cc:492] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:42.247381 3912 ts_tablet_manager.cc:1367] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:09:42.247800 3912 raft_consensus.cc:348] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:42.247905 3912 raft_consensus.cc:374] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.247934 3912 raft_consensus.cc:729] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.248050 3912 consensus_queue.cc:260] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:42.248298 3910 tablet_bootstrap.cc:654] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.248407 3912 ts_tablet_manager.cc:1398] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:42.250310 3922 tablet_bootstrap.cc:654] T 22bb82c5d65640128fc3271212aafb60 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.251523 3910 tablet_bootstrap.cc:492] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:42.251600 3910 ts_tablet_manager.cc:1367] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:09:42.252004 3910 raft_consensus.cc:348] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:42.252099 3910 raft_consensus.cc:374] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.252130 3910 raft_consensus.cc:729] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.252283 3910 consensus_queue.cc:260] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:42.252380 29871 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:42.252501 29871 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:42.252844 3910 ts_tablet_manager.cc:1398] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:42.253732 3922 tablet_bootstrap.cc:492] T 22bb82c5d65640128fc3271212aafb60 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:42.253947 3922 ts_tablet_manager.cc:1367] T 22bb82c5d65640128fc3271212aafb60 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:42.254457 3922 raft_consensus.cc:348] T 22bb82c5d65640128fc3271212aafb60 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:42.254616 3922 raft_consensus.cc:374] T 22bb82c5d65640128fc3271212aafb60 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.254817 3922 raft_consensus.cc:729] T 22bb82c5d65640128fc3271212aafb60 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.255133 3922 consensus_queue.cc:260] T 22bb82c5d65640128fc3271212aafb60 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:42.255429 29871 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 331e53e2abef4e43a9fbe4e6f4b67229
I20240501 14:09:42.256126 32665 tablet_service.cc:1514] Processing DeleteTablet for tablet 331e53e2abef4e43a9fbe4e6f4b67229 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:42 UTC) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:42.256574 32534 tablet_service.cc:1514] Processing DeleteTablet for tablet 331e53e2abef4e43a9fbe4e6f4b67229 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:42 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:42.256826 327 tablet_service.cc:1514] Processing DeleteTablet for tablet 331e53e2abef4e43a9fbe4e6f4b67229 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:42 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:42.256889 3922 ts_tablet_manager.cc:1398] T 22bb82c5d65640128fc3271212aafb60 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:42.257110 3929 tablet_replica.cc:330] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:42.257238 3929 raft_consensus.cc:2229] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:42.257349 3929 raft_consensus.cc:2258] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:42.257572 3928 tablet_replica.cc:330] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:42.257709 3928 raft_consensus.cc:2229] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:42.257836 3928 raft_consensus.cc:2258] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:42.258313 3929 ts_tablet_manager.cc:1875] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:42.259775 3930 tablet_replica.cc:330] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:42.259912 3930 raft_consensus.cc:2229] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:42.259977 30146 raft_consensus.cc:1203] T 776d6722a14345f490f69f90d80d6a6a P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:42.260084 3930 raft_consensus.cc:2258] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:42.261565 3928 ts_tablet_manager.cc:1875] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:42.263532 3930 ts_tablet_manager.cc:1875] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:42.264088 3928 ts_tablet_manager.cc:1888] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:42.264294 3928 log.cc:1196] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/331e53e2abef4e43a9fbe4e6f4b67229
I20240501 14:09:42.264665 3928 ts_tablet_manager.cc:1909] T 331e53e2abef4e43a9fbe4e6f4b67229 P db935501e9104de5b15c8780681d6c38: Deleting consensus metadata
I20240501 14:09:42.264895 3929 ts_tablet_manager.cc:1888] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:42.265439 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet 331e53e2abef4e43a9fbe4e6f4b67229 (table rebalancer_dynamic_table_new_name [id=8d598c3820f7442b87a92f24c4f44125]) successfully deleted
I20240501 14:09:42.266188 3916 tablet_bootstrap.cc:492] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:42.266274 3916 ts_tablet_manager.cc:1367] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
I20240501 14:09:42.266721 3916 raft_consensus.cc:348] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:42.266839 3916 raft_consensus.cc:374] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.266878 3916 raft_consensus.cc:729] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.267024 3916 consensus_queue.cc:260] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:42.267086 3930 ts_tablet_manager.cc:1888] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:42.267154 3930 log.cc:1196] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/331e53e2abef4e43a9fbe4e6f4b67229
I20240501 14:09:42.267360 3916 ts_tablet_manager.cc:1398] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:42.267418 3930 ts_tablet_manager.cc:1909] T 331e53e2abef4e43a9fbe4e6f4b67229 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:09:42.268476 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 331e53e2abef4e43a9fbe4e6f4b67229 (table rebalancer_dynamic_table_new_name [id=8d598c3820f7442b87a92f24c4f44125]) successfully deleted
I20240501 14:09:42.265014 3929 log.cc:1196] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/331e53e2abef4e43a9fbe4e6f4b67229
I20240501 14:09:42.269354 3929 ts_tablet_manager.cc:1909] T 331e53e2abef4e43a9fbe4e6f4b67229 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:42.270051 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet 331e53e2abef4e43a9fbe4e6f4b67229 (table rebalancer_dynamic_table_new_name [id=8d598c3820f7442b87a92f24c4f44125]) successfully deleted
I20240501 14:09:42.277824 3042 raft_consensus.cc:482] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:42.277943 3042 raft_consensus.cc:504] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:42.278373 3042 leader_election.cc:288] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:42.278640 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6229c0c3e98c462b95f9bfe066367e87" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:42.278685 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6229c0c3e98c462b95f9bfe066367e87" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:42.278777 32553 raft_consensus.cc:2454] T 6229c0c3e98c462b95f9bfe066367e87 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:42.278827 348 raft_consensus.cc:2454] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:42.279094 32621 leader_election.cc:302] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:42.279263 3042 raft_consensus.cc:2781] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:42.279318 3042 raft_consensus.cc:482] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:42.279345 3042 raft_consensus.cc:3037] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.280160 3042 raft_consensus.cc:504] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:42.280508 3042 leader_election.cc:288] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:42.280800 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6229c0c3e98c462b95f9bfe066367e87" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:42.280908 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6229c0c3e98c462b95f9bfe066367e87" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:42.280939 348 raft_consensus.cc:3037] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.280995 32553 raft_consensus.cc:3037] T 6229c0c3e98c462b95f9bfe066367e87 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.281960 32553 raft_consensus.cc:2454] T 6229c0c3e98c462b95f9bfe066367e87 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:42.282037 348 raft_consensus.cc:2454] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:42.282387 32621 leader_election.cc:302] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:42.282543 3042 raft_consensus.cc:2781] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:42.282711 3042 raft_consensus.cc:686] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Becoming Leader. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
I20240501 14:09:42.282845 3042 consensus_queue.cc:237] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:42.284883 29871 catalog_manager.cc:5484] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db935501e9104de5b15c8780681d6c38 (127.22.252.135). New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:42.304010 3755 raft_consensus.cc:482] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:42.304118 3755 raft_consensus.cc:504] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:42.304509 3755 leader_election.cc:288] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:42.304832 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22bb82c5d65640128fc3271212aafb60" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:42.304973 32416 raft_consensus.cc:2454] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:42.305239 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22bb82c5d65640128fc3271212aafb60" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:42.305290 32489 leader_election.cc:302] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:42.305392 30406 raft_consensus.cc:2454] T 22bb82c5d65640128fc3271212aafb60 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:42.305526 3755 raft_consensus.cc:2781] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:42.305584 3755 raft_consensus.cc:482] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:42.305621 3755 raft_consensus.cc:3037] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.306782 3755 raft_consensus.cc:504] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:42.307117 3755 leader_election.cc:288] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:42.307408 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22bb82c5d65640128fc3271212aafb60" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:42.307507 30406 raft_consensus.cc:3037] T 22bb82c5d65640128fc3271212aafb60 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.307554 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22bb82c5d65640128fc3271212aafb60" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:42.307670 32416 raft_consensus.cc:3037] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.308411 30406 raft_consensus.cc:2454] T 22bb82c5d65640128fc3271212aafb60 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:42.308662 32487 leader_election.cc:302] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:42.308691 32416 raft_consensus.cc:2454] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:42.308982 3755 raft_consensus.cc:2781] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:42.309128 3755 raft_consensus.cc:686] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:09:42.309265 3755 consensus_queue.cc:237] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:42.311399 29871 catalog_manager.cc:5484] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:42.335808 32685 raft_consensus.cc:1203] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:42.357044 29870 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:42.360252 3755 raft_consensus.cc:482] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:42.360368 3755 raft_consensus.cc:504] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:42.360836 3755 leader_election.cc:288] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:42.361197 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63e9bcd27234f2f9a9e0b268fd73486" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:42.361351 348 raft_consensus.cc:2454] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:42.361393 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63e9bcd27234f2f9a9e0b268fd73486" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:42.361516 30264 raft_consensus.cc:2454] T a63e9bcd27234f2f9a9e0b268fd73486 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:42.361653 32489 leader_election.cc:302] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:42.361804 3755 raft_consensus.cc:2781] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:42.361860 3755 raft_consensus.cc:482] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:42.361904 3755 raft_consensus.cc:3037] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.362665 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd6f3226c7c49d7b68fe8da09e288d9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=11b95b09c0a54d8db59b840ee4096ff2]), partition=
I20240501 14:09:42.362869 3755 raft_consensus.cc:504] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:42.362911 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd6f3226c7c49d7b68fe8da09e288d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.363164 3755 leader_election.cc:288] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:42.363413 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63e9bcd27234f2f9a9e0b268fd73486" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:42.363485 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd6f3226c7c49d7b68fe8da09e288d9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=11b95b09c0a54d8db59b840ee4096ff2]), partition=
I20240501 14:09:42.363512 30264 raft_consensus.cc:3037] T a63e9bcd27234f2f9a9e0b268fd73486 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.363533 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63e9bcd27234f2f9a9e0b268fd73486" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:42.363619 348 raft_consensus.cc:3037] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.363708 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd6f3226c7c49d7b68fe8da09e288d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.364526 30264 raft_consensus.cc:2454] T a63e9bcd27234f2f9a9e0b268fd73486 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:42.364634 348 raft_consensus.cc:2454] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:42.365020 32489 leader_election.cc:302] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:42.365164 3916 tablet_bootstrap.cc:492] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:42.365979 3916 tablet_bootstrap.cc:654] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.366166 3755 raft_consensus.cc:2781] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:42.366243 3755 raft_consensus.cc:686] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:09:42.366376 3755 consensus_queue.cc:237] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:42.366729 3914 tablet_bootstrap.cc:492] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:42.367501 3914 tablet_bootstrap.cc:654] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.368145 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd6f3226c7c49d7b68fe8da09e288d9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=11b95b09c0a54d8db59b840ee4096ff2]), partition=
I20240501 14:09:42.368371 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd6f3226c7c49d7b68fe8da09e288d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.368476 29870 catalog_manager.cc:5484] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:42.370095 3910 tablet_bootstrap.cc:492] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:42.370935 3910 tablet_bootstrap.cc:654] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.371445 3914 tablet_bootstrap.cc:492] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:42.371528 3914 ts_tablet_manager.cc:1367] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:42.371974 3914 raft_consensus.cc:348] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:42.372064 3914 raft_consensus.cc:374] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.372102 3914 raft_consensus.cc:729] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.372155 3910 tablet_bootstrap.cc:492] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:42.372274 3910 ts_tablet_manager.cc:1367] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:42.372247 3914 consensus_queue.cc:260] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:42.372517 3914 ts_tablet_manager.cc:1398] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:42.372715 3910 raft_consensus.cc:348] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:42.372807 3910 raft_consensus.cc:374] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.372839 3910 raft_consensus.cc:729] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.372965 3916 tablet_bootstrap.cc:492] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:42.372944 3910 consensus_queue.cc:260] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:42.373040 3916 ts_tablet_manager.cc:1367] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:09:42.373205 3910 ts_tablet_manager.cc:1398] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:42.373484 3916 raft_consensus.cc:348] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:42.373584 3916 raft_consensus.cc:374] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.373620 3916 raft_consensus.cc:729] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.373731 3916 consensus_queue.cc:260] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:42.374002 3916 ts_tablet_manager.cc:1398] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:42.391949 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:42.394026 29993 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:42.399436 30146 raft_consensus.cc:1203] T 575e8cde863540b09a52fe590cbf2bfc P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:42.400465 30124 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:42.404884 30388 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:42.407153 327 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:42.407167 32400 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:42.426257 348 raft_consensus.cc:1203] T 7e1895c0656d49c09266cdbd231497cb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:42.432597 32665 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:42.433220 32534 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:42.452849 348 raft_consensus.cc:1203] T ee998c3fcf3449d5845ac1c950f326d7 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:42.459270 30264 raft_consensus.cc:1203] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:42.502358 32416 consensus_queue.cc:237] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:42.503803 32553 raft_consensus.cc:1261] T 776d6722a14345f490f69f90d80d6a6a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:42.503813 30146 raft_consensus.cc:1261] T 776d6722a14345f490f69f90d80d6a6a P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:42.504287 2503 consensus_queue.cc:1035] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:42.504652 3579 consensus_queue.cc:1035] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:42.506243 3579 raft_consensus.cc:2932] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:42.506623 30146 raft_consensus.cc:2932] T 776d6722a14345f490f69f90d80d6a6a P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:42.509315 32553 raft_consensus.cc:2932] T 776d6722a14345f490f69f90d80d6a6a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:42.509341 29867 catalog_manager.cc:5484] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:42.513288 327 tablet_service.cc:1514] Processing DeleteTablet for tablet 776d6722a14345f490f69f90d80d6a6a with delete_type TABLET_DATA_TOMBSTONED (TS 9ecd907cf5634ac98f3f54f3dbec6849 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:42.513500 3930 tablet_replica.cc:330] T 776d6722a14345f490f69f90d80d6a6a P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:42.513648 3930 raft_consensus.cc:2229] T 776d6722a14345f490f69f90d80d6a6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:42.513764 3930 raft_consensus.cc:2258] T 776d6722a14345f490f69f90d80d6a6a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:42.514195 3930 ts_tablet_manager.cc:1875] T 776d6722a14345f490f69f90d80d6a6a P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:42.514429 3579 raft_consensus.cc:482] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:42.514520 3579 raft_consensus.cc:504] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:42.514923 3579 leader_election.cc:288] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:42.515291 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd6f3226c7c49d7b68fe8da09e288d9" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:42.515307 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd6f3226c7c49d7b68fe8da09e288d9" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:42.515431 32685 raft_consensus.cc:2454] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:42.515429 32553 raft_consensus.cc:2454] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:42.515755 32356 leader_election.cc:302] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:42.515906 3579 raft_consensus.cc:2781] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:42.515949 3579 raft_consensus.cc:482] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:42.516006 3579 raft_consensus.cc:3037] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.516933 3579 raft_consensus.cc:504] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:42.516188 3930 ts_tablet_manager.cc:1888] T 776d6722a14345f490f69f90d80d6a6a P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:09:42.517632 3930 log.cc:1196] T 776d6722a14345f490f69f90d80d6a6a P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/776d6722a14345f490f69f90d80d6a6a
I20240501 14:09:42.517598 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd6f3226c7c49d7b68fe8da09e288d9" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:42.517697 32685 raft_consensus.cc:3037] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.517683 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd6f3226c7c49d7b68fe8da09e288d9" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:42.517794 32553 raft_consensus.cc:3037] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.518747 32685 raft_consensus.cc:2454] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:42.518935 32553 raft_consensus.cc:2454] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:42.519047 32356 leader_election.cc:302] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:42.519229 2503 raft_consensus.cc:2781] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:42.519307 2503 raft_consensus.cc:686] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:42.519295 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 776d6722a14345f490f69f90d80d6a6a (table rebalancer_extra_table_6 [id=d23b9f10156c4074a707699cf10e3b55]) successfully deleted
I20240501 14:09:42.519428 2503 consensus_queue.cc:237] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:42.521000 3579 leader_election.cc:288] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:42.521677 29864 catalog_manager.cc:5484] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:42.627975 29871 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:42.629078 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:42.633419 32553 raft_consensus.cc:1261] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:42.633493 32685 raft_consensus.cc:1261] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:42.633858 4010 consensus_queue.cc:1035] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:42.634296 4010 consensus_queue.cc:1035] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:42.636590 29871 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet 6cd6f3226c7c49d7b68fe8da09e288d9 (table rebalancer_dynamic_table [id=11b95b09c0a54d8db59b840ee4096ff2]). Expected version 1 got 0
I20240501 14:09:42.638108 29871 catalog_manager.cc:5576] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497) does not have the latest schema for tablet 6cd6f3226c7c49d7b68fe8da09e288d9 (table rebalancer_dynamic_table [id=11b95b09c0a54d8db59b840ee4096ff2]). Expected version 1 got 0
I20240501 14:09:42.638397 3822 tablet.cc:1722] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:42.638713 3822 tablet.cc:1620] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:42.639412 4066 tablet.cc:1722] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:42.643021 29871 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=11b95b09c0a54d8db59b840ee4096ff2] alter complete (version 1)
I20240501 14:09:42.643594 4066 tablet.cc:1620] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:42.654088 29993 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:42.657979 32534 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:42.661594 327 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:42.665253 30388 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:42.679102 32400 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:42.689368 30124 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:42.690021 32665 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:42.713742 3755 consensus_queue.cc:1035] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:42.717545 3942 consensus_queue.cc:1035] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:42.729947 3042 consensus_queue.cc:1035] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:42.741796 3755 consensus_queue.cc:1035] T 22bb82c5d65640128fc3271212aafb60 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:42.745098 29866 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet 6cd6f3226c7c49d7b68fe8da09e288d9 (table rebalancer_dynamic_table [id=11b95b09c0a54d8db59b840ee4096ff2]). Expected version 1 got 0
I20240501 14:09:42.749006 29871 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:42.749590 4075 tablet.cc:1722] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:42.749926 4075 tablet.cc:1620] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:42.755177 4066 tablet.cc:1722] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:42.755483 4066 tablet.cc:1620] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:42.756494 3822 tablet.cc:1722] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:42.756755 3822 tablet.cc:1620] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:42.763828 29870 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=11b95b09c0a54d8db59b840ee4096ff2] alter complete (version 2)
I20240501 14:09:42.767650 29870 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet 6cd6f3226c7c49d7b68fe8da09e288d9 (table rebalancer_dynamic_table [id=11b95b09c0a54d8db59b840ee4096ff2]). Expected version 2 got 1
I20240501 14:09:42.768823 4075 tablet.cc:1722] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:42.769176 4075 tablet.cc:1620] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:42.812691 30146 consensus_queue.cc:237] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:42.813925 348 raft_consensus.cc:1261] T ee998c3fcf3449d5845ac1c950f326d7 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:42.814013 30013 raft_consensus.cc:1261] T ee998c3fcf3449d5845ac1c950f326d7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:42.814414 2960 consensus_queue.cc:1035] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:42.814648 2960 consensus_queue.cc:1035] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:42.816349 2960 raft_consensus.cc:2932] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:42.818951 30013 raft_consensus.cc:2932] T ee998c3fcf3449d5845ac1c950f326d7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:42.819272 348 raft_consensus.cc:2932] T ee998c3fcf3449d5845ac1c950f326d7 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:42.819298 29870 catalog_manager.cc:5484] T ee998c3fcf3449d5845ac1c950f326d7 P d35c16dcb506426993811bb2519d7db3 reported cstate change: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:42.822062 30256 tablet_service.cc:1514] Processing DeleteTablet for tablet ee998c3fcf3449d5845ac1c950f326d7 with delete_type TABLET_DATA_TOMBSTONED (TS 1d0bb69611af4876a2b3b33c4766e13a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:42.822397 4095 tablet_replica.cc:330] T ee998c3fcf3449d5845ac1c950f326d7 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:42.822592 4095 raft_consensus.cc:2229] T ee998c3fcf3449d5845ac1c950f326d7 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:42.822813 4095 raft_consensus.cc:2258] T ee998c3fcf3449d5845ac1c950f326d7 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:42.823513 4095 ts_tablet_manager.cc:1875] T ee998c3fcf3449d5845ac1c950f326d7 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:42.826825 4095 ts_tablet_manager.cc:1888] T ee998c3fcf3449d5845ac1c950f326d7 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:42.826938 4095 log.cc:1196] T ee998c3fcf3449d5845ac1c950f326d7 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/ee998c3fcf3449d5845ac1c950f326d7
I20240501 14:09:42.827728 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet ee998c3fcf3449d5845ac1c950f326d7 (table rebalance_test_table_5 [id=440878c63b0f4706b34fd42316991270]) successfully deleted
I20240501 14:09:42.844801 3755 consensus_queue.cc:1035] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:42.848559 3954 consensus_queue.cc:1035] T a63e9bcd27234f2f9a9e0b268fd73486 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:42.869889 29870 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:42.876181 4066 tablet.cc:1722] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:42.876633 3822 tablet.cc:1722] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:42.877292 4075 tablet.cc:1722] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:42.880280 29868 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:42.882792 29869 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=11b95b09c0a54d8db59b840ee4096ff2] alter complete (version 3)
I20240501 14:09:42.888198 327 tablet_service.cc:1467] Processing CreateTablet for tablet bf0712a5e11c46a291b90bfb491b8b25 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=fa42a548a7f24eab9694087203fda9ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.888422 32402 tablet_service.cc:1467] Processing CreateTablet for tablet 0ff4979d0b8b428a88c0eca25e018e86 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=fa42a548a7f24eab9694087203fda9ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.888505 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf0712a5e11c46a291b90bfb491b8b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.888665 32402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ff4979d0b8b428a88c0eca25e018e86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.888736 32400 tablet_service.cc:1467] Processing CreateTablet for tablet bf0712a5e11c46a291b90bfb491b8b25 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=fa42a548a7f24eab9694087203fda9ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.888931 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf0712a5e11c46a291b90bfb491b8b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.890949 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 0ff4979d0b8b428a88c0eca25e018e86 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=fa42a548a7f24eab9694087203fda9ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.891220 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ff4979d0b8b428a88c0eca25e018e86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.892144 327 tablet_service.cc:1467] Processing CreateTablet for tablet 7628284cb4484ee5b87173790179def0 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=fa42a548a7f24eab9694087203fda9ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.892320 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 7628284cb4484ee5b87173790179def0 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=fa42a548a7f24eab9694087203fda9ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.892547 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7628284cb4484ee5b87173790179def0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.893153 4141 tablet_bootstrap.cc:492] T bf0712a5e11c46a291b90bfb491b8b25 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:42.893263 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7628284cb4484ee5b87173790179def0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.894546 4141 tablet_bootstrap.cc:654] T bf0712a5e11c46a291b90bfb491b8b25 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.895469 32665 tablet_service.cc:1467] Processing CreateTablet for tablet bf0712a5e11c46a291b90bfb491b8b25 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=fa42a548a7f24eab9694087203fda9ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.895664 4141 tablet_bootstrap.cc:492] T bf0712a5e11c46a291b90bfb491b8b25 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:42.895711 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf0712a5e11c46a291b90bfb491b8b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.895737 4141 ts_tablet_manager.cc:1367] T bf0712a5e11c46a291b90bfb491b8b25 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:42.896162 4141 raft_consensus.cc:348] T bf0712a5e11c46a291b90bfb491b8b25 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:42.896272 4141 raft_consensus.cc:374] T bf0712a5e11c46a291b90bfb491b8b25 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.896306 4141 raft_consensus.cc:729] T bf0712a5e11c46a291b90bfb491b8b25 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.896433 4141 consensus_queue.cc:260] T bf0712a5e11c46a291b90bfb491b8b25 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:42.896690 4141 ts_tablet_manager.cc:1398] T bf0712a5e11c46a291b90bfb491b8b25 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:42.896833 4141 tablet_bootstrap.cc:492] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:42.897598 4141 tablet_bootstrap.cc:654] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.897890 4144 tablet_bootstrap.cc:492] T bf0712a5e11c46a291b90bfb491b8b25 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:42.898929 4144 tablet_bootstrap.cc:654] T bf0712a5e11c46a291b90bfb491b8b25 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.898897 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 0ff4979d0b8b428a88c0eca25e018e86 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=fa42a548a7f24eab9694087203fda9ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.899097 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 7628284cb4484ee5b87173790179def0 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=fa42a548a7f24eab9694087203fda9ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.899158 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ff4979d0b8b428a88c0eca25e018e86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.901548 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7628284cb4484ee5b87173790179def0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.902634 4140 tablet_bootstrap.cc:492] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:42.902861 4142 tablet_bootstrap.cc:492] T 0ff4979d0b8b428a88c0eca25e018e86 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:42.903832 4142 tablet_bootstrap.cc:654] T 0ff4979d0b8b428a88c0eca25e018e86 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.904752 4140 tablet_bootstrap.cc:654] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.906363 4141 tablet_bootstrap.cc:492] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:42.906446 4141 ts_tablet_manager.cc:1367] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 14:09:42.906489 4140 tablet_bootstrap.cc:492] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:42.906560 4140 ts_tablet_manager.cc:1367] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:42.906845 4141 raft_consensus.cc:348] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:42.906944 4141 raft_consensus.cc:374] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.906891 4140 raft_consensus.cc:348] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:42.906975 4141 raft_consensus.cc:729] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.906991 4140 raft_consensus.cc:374] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.907035 4140 raft_consensus.cc:729] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.907167 4140 consensus_queue.cc:260] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:42.907167 4141 consensus_queue.cc:260] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:42.907680 4141 ts_tablet_manager.cc:1398] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:42.907825 4141 tablet_bootstrap.cc:492] T 7628284cb4484ee5b87173790179def0 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:42.908274 4140 ts_tablet_manager.cc:1398] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:42.908473 4140 tablet_bootstrap.cc:492] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:42.908720 4141 tablet_bootstrap.cc:654] T 7628284cb4484ee5b87173790179def0 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.911427 4140 tablet_bootstrap.cc:654] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.912899 4142 tablet_bootstrap.cc:492] T 0ff4979d0b8b428a88c0eca25e018e86 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:42.913012 4142 ts_tablet_manager.cc:1367] T 0ff4979d0b8b428a88c0eca25e018e86 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:42.913537 4144 tablet_bootstrap.cc:492] T bf0712a5e11c46a291b90bfb491b8b25 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:42.913522 4142 raft_consensus.cc:348] T 0ff4979d0b8b428a88c0eca25e018e86 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:42.913641 4142 raft_consensus.cc:374] T 0ff4979d0b8b428a88c0eca25e018e86 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.913676 4142 raft_consensus.cc:729] T 0ff4979d0b8b428a88c0eca25e018e86 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.913820 4144 ts_tablet_manager.cc:1367] T bf0712a5e11c46a291b90bfb491b8b25 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:09:42.913805 4142 consensus_queue.cc:260] T 0ff4979d0b8b428a88c0eca25e018e86 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:42.914093 4142 ts_tablet_manager.cc:1398] T 0ff4979d0b8b428a88c0eca25e018e86 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:42.914677 4144 raft_consensus.cc:348] T bf0712a5e11c46a291b90bfb491b8b25 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:42.914788 4144 raft_consensus.cc:374] T bf0712a5e11c46a291b90bfb491b8b25 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.914827 4144 raft_consensus.cc:729] T bf0712a5e11c46a291b90bfb491b8b25 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.914976 4144 consensus_queue.cc:260] T bf0712a5e11c46a291b90bfb491b8b25 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:42.916025 4140 tablet_bootstrap.cc:492] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:42.916180 4140 ts_tablet_manager.cc:1367] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:42.916803 4140 raft_consensus.cc:348] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:42.916980 4140 raft_consensus.cc:374] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.917060 4140 raft_consensus.cc:729] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.917239 4140 consensus_queue.cc:260] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:42.918375 4140 ts_tablet_manager.cc:1398] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:42.919622 4144 ts_tablet_manager.cc:1398] T bf0712a5e11c46a291b90bfb491b8b25 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:42.919811 4144 tablet_bootstrap.cc:492] T 0ff4979d0b8b428a88c0eca25e018e86 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:42.920907 4144 tablet_bootstrap.cc:654] T 0ff4979d0b8b428a88c0eca25e018e86 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.923949 4141 tablet_bootstrap.cc:492] T 7628284cb4484ee5b87173790179def0 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:42.924078 4141 ts_tablet_manager.cc:1367] T 7628284cb4484ee5b87173790179def0 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:09:42.925156 4141 raft_consensus.cc:348] T 7628284cb4484ee5b87173790179def0 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:42.925355 4141 raft_consensus.cc:374] T 7628284cb4484ee5b87173790179def0 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.925570 4141 raft_consensus.cc:729] T 7628284cb4484ee5b87173790179def0 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.925752 4141 consensus_queue.cc:260] T 7628284cb4484ee5b87173790179def0 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:42.926162 4141 ts_tablet_manager.cc:1398] T 7628284cb4484ee5b87173790179def0 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:42.927704 3126 raft_consensus.cc:482] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:42.927783 3126 raft_consensus.cc:504] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:42.928184 3126 leader_election.cc:288] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:42.928422 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0712a5e11c46a291b90bfb491b8b25" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:42.928546 32685 raft_consensus.cc:2454] T bf0712a5e11c46a291b90bfb491b8b25 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:42.928823 32750 leader_election.cc:302] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:42.929153 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0712a5e11c46a291b90bfb491b8b25" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:42.929272 32416 raft_consensus.cc:2454] T bf0712a5e11c46a291b90bfb491b8b25 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:42.929886 3126 raft_consensus.cc:2781] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:42.929944 3126 raft_consensus.cc:482] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:42.929989 3126 raft_consensus.cc:3037] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.930969 3126 raft_consensus.cc:504] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:42.931288 3126 leader_election.cc:288] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:42.931823 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0712a5e11c46a291b90bfb491b8b25" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:42.931914 32416 raft_consensus.cc:3037] T bf0712a5e11c46a291b90bfb491b8b25 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.932919 32416 raft_consensus.cc:2454] T bf0712a5e11c46a291b90bfb491b8b25 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:42.933339 32752 leader_election.cc:302] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:42.933792 3126 raft_consensus.cc:2781] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:42.933930 3126 raft_consensus.cc:686] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:42.934217 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0712a5e11c46a291b90bfb491b8b25" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:42.934338 32685 raft_consensus.cc:3037] T bf0712a5e11c46a291b90bfb491b8b25 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.934216 3126 consensus_queue.cc:237] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:42.935408 32685 raft_consensus.cc:2454] T bf0712a5e11c46a291b90bfb491b8b25 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:42.935520 4144 tablet_bootstrap.cc:492] T 0ff4979d0b8b428a88c0eca25e018e86 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:42.935616 4144 ts_tablet_manager.cc:1367] T 0ff4979d0b8b428a88c0eca25e018e86 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:09:42.936038 4144 raft_consensus.cc:348] T 0ff4979d0b8b428a88c0eca25e018e86 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:42.936146 4144 raft_consensus.cc:374] T 0ff4979d0b8b428a88c0eca25e018e86 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.936213 4144 raft_consensus.cc:729] T 0ff4979d0b8b428a88c0eca25e018e86 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.936409 4144 consensus_queue.cc:260] T 0ff4979d0b8b428a88c0eca25e018e86 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:42.937609 4144 ts_tablet_manager.cc:1398] T 0ff4979d0b8b428a88c0eca25e018e86 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:42.937868 4144 tablet_bootstrap.cc:492] T 7628284cb4484ee5b87173790179def0 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:42.939010 4144 tablet_bootstrap.cc:654] T 7628284cb4484ee5b87173790179def0 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.939723 29864 catalog_manager.cc:5484] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:42.946285 4144 tablet_bootstrap.cc:492] T 7628284cb4484ee5b87173790179def0 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:42.947372 4144 ts_tablet_manager.cc:1367] T 7628284cb4484ee5b87173790179def0 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:42.952926 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:42.954161 29993 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:42.953728 4144 raft_consensus.cc:348] T 7628284cb4484ee5b87173790179def0 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:42.955339 4144 raft_consensus.cc:374] T 7628284cb4484ee5b87173790179def0 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.955543 4144 raft_consensus.cc:729] T 7628284cb4484ee5b87173790179def0 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.955739 4144 consensus_queue.cc:260] T 7628284cb4484ee5b87173790179def0 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:42.957158 4144 ts_tablet_manager.cc:1398] T 7628284cb4484ee5b87173790179def0 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240501 14:09:42.959930 30388 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:42.973050 30124 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:42.973932 32664 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:42.984303 32400 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:42.984530 327 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:42.985384 32534 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:43.019280 29865 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:43.019439 29865 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:43.021770 29865 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 6cd6f3226c7c49d7b68fe8da09e288d9
I20240501 14:09:43.022280 32400 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cd6f3226c7c49d7b68fe8da09e288d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:43 UTC) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:43.022330 32534 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cd6f3226c7c49d7b68fe8da09e288d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:43 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:43.022778 32664 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cd6f3226c7c49d7b68fe8da09e288d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:43 UTC) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:43.023108 4170 tablet_replica.cc:330] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:43.024194 4170 raft_consensus.cc:2229] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:43.024394 4170 raft_consensus.cc:2258] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:43.025225 4170 ts_tablet_manager.cc:1875] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:43.027415 4170 ts_tablet_manager.cc:1888] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:43.027484 4170 log.cc:1196] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/6cd6f3226c7c49d7b68fe8da09e288d9
I20240501 14:09:43.027731 4170 ts_tablet_manager.cc:1909] T 6cd6f3226c7c49d7b68fe8da09e288d9 P db935501e9104de5b15c8780681d6c38: Deleting consensus metadata
I20240501 14:09:43.028367 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet 6cd6f3226c7c49d7b68fe8da09e288d9 (table rebalancer_dynamic_table_new_name [id=11b95b09c0a54d8db59b840ee4096ff2]) successfully deleted
I20240501 14:09:43.047525 4168 tablet_replica.cc:330] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:43.047698 4168 raft_consensus.cc:2229] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:43.047837 4168 raft_consensus.cc:2258] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:43.048557 4168 ts_tablet_manager.cc:1875] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:43.050668 4168 ts_tablet_manager.cc:1888] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:43.050750 4168 log.cc:1196] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/6cd6f3226c7c49d7b68fe8da09e288d9
I20240501 14:09:43.051039 4168 ts_tablet_manager.cc:1909] T 6cd6f3226c7c49d7b68fe8da09e288d9 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:43.051684 4169 tablet_replica.cc:330] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:43.051836 4169 raft_consensus.cc:2229] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:43.052093 4169 raft_consensus.cc:2258] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:43.052654 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet 6cd6f3226c7c49d7b68fe8da09e288d9 (table rebalancer_dynamic_table_new_name [id=11b95b09c0a54d8db59b840ee4096ff2]) successfully deleted
I20240501 14:09:43.052713 4169 ts_tablet_manager.cc:1875] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:43.054550 4169 ts_tablet_manager.cc:1888] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:43.054616 4169 log.cc:1196] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/6cd6f3226c7c49d7b68fe8da09e288d9
I20240501 14:09:43.054981 4169 ts_tablet_manager.cc:1909] T 6cd6f3226c7c49d7b68fe8da09e288d9 P a91029b90bf9415a89a2f70736759ad3: Deleting consensus metadata
I20240501 14:09:43.055510 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 6cd6f3226c7c49d7b68fe8da09e288d9 (table rebalancer_dynamic_table_new_name [id=11b95b09c0a54d8db59b840ee4096ff2]) successfully deleted
I20240501 14:09:43.057220 30146 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c362bd01a59b4bcf85eeb59713e58b64"
dest_uuid: "d35c16dcb506426993811bb2519d7db3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50880
I20240501 14:09:43.057350 30146 raft_consensus.cc:595] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:43.084673 3579 raft_consensus.cc:482] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:43.084782 3579 raft_consensus.cc:504] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:43.085104 3579 leader_election.cc:288] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:43.085417 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff4979d0b8b428a88c0eca25e018e86" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:43.085556 32685 raft_consensus.cc:2454] T 0ff4979d0b8b428a88c0eca25e018e86 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:43.085549 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff4979d0b8b428a88c0eca25e018e86" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:43.085692 30013 raft_consensus.cc:2454] T 0ff4979d0b8b428a88c0eca25e018e86 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:43.085985 32358 leader_election.cc:302] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:43.086337 3579 raft_consensus.cc:2781] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:43.086411 3579 raft_consensus.cc:482] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:43.086448 3579 raft_consensus.cc:3037] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.087473 3579 raft_consensus.cc:504] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:43.087795 3579 leader_election.cc:288] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:43.087807 4152 raft_consensus.cc:482] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:43.087905 4152 raft_consensus.cc:504] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:43.088311 4152 leader_election.cc:288] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:43.088476 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff4979d0b8b428a88c0eca25e018e86" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:43.088563 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff4979d0b8b428a88c0eca25e018e86" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:43.088563 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7628284cb4484ee5b87173790179def0" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:43.088685 30013 raft_consensus.cc:3037] T 0ff4979d0b8b428a88c0eca25e018e86 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.088694 32683 raft_consensus.cc:2454] T 7628284cb4484ee5b87173790179def0 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:43.088574 32685 raft_consensus.cc:3037] T 0ff4979d0b8b428a88c0eca25e018e86 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.089615 30013 raft_consensus.cc:2454] T 0ff4979d0b8b428a88c0eca25e018e86 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:43.089794 32685 raft_consensus.cc:2454] T 0ff4979d0b8b428a88c0eca25e018e86 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:43.089900 32750 leader_election.cc:302] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:43.090073 4152 raft_consensus.cc:2781] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:43.090129 4152 raft_consensus.cc:482] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:43.090163 4152 raft_consensus.cc:3037] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.090170 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7628284cb4484ee5b87173790179def0" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:43.090283 32416 raft_consensus.cc:2454] T 7628284cb4484ee5b87173790179def0 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:43.091007 2960 raft_consensus.cc:979] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3: : Instructing follower d1c217469ae4478ebd49e4ed9f7352d7 to start an election
I20240501 14:09:43.091096 2960 raft_consensus.cc:1067] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Signalling peer d1c217469ae4478ebd49e4ed9f7352d7 to start an election
I20240501 14:09:43.091130 4152 raft_consensus.cc:504] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:43.091526 4152 leader_election.cc:288] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:43.091671 32356 leader_election.cc:302] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:43.092096 30406 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c362bd01a59b4bcf85eeb59713e58b64"
dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
from {username='slave'} at 127.22.252.130:46573
I20240501 14:09:43.092170 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7628284cb4484ee5b87173790179def0" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:43.092276 32685 raft_consensus.cc:3037] T 7628284cb4484ee5b87173790179def0 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.092768 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7628284cb4484ee5b87173790179def0" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:43.092855 32416 raft_consensus.cc:3037] T 7628284cb4484ee5b87173790179def0 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.093194 32685 raft_consensus.cc:2454] T 7628284cb4484ee5b87173790179def0 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:43.093497 32750 leader_election.cc:302] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:43.093683 4152 raft_consensus.cc:2781] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:43.093761 4152 raft_consensus.cc:686] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:43.093837 32416 raft_consensus.cc:2454] T 7628284cb4484ee5b87173790179def0 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:43.093888 4152 consensus_queue.cc:237] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:43.094241 30406 raft_consensus.cc:482] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:43.094321 30406 raft_consensus.cc:3037] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:43.094390 4010 raft_consensus.cc:2781] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:43.094478 4010 raft_consensus.cc:686] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:43.094588 4010 consensus_queue.cc:237] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:43.095270 30406 raft_consensus.cc:504] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:43.095777 30406 leader_election.cc:288] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:43.095863 29871 catalog_manager.cc:5484] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:43.096048 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c362bd01a59b4bcf85eeb59713e58b64" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:43.096174 30264 raft_consensus.cc:3037] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:43.097311 30264 raft_consensus.cc:2454] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 2.
I20240501 14:09:43.097326 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c362bd01a59b4bcf85eeb59713e58b64" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:43.097445 30146 raft_consensus.cc:3032] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:43.097486 30146 raft_consensus.cc:729] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:43.097628 30146 consensus_queue.cc:260] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:43.097828 30146 raft_consensus.cc:3037] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:43.098114 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c362bd01a59b4bcf85eeb59713e58b64" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:43.098233 32685 raft_consensus.cc:3037] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:43.098848 30146 raft_consensus.cc:2454] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 2.
I20240501 14:09:43.099102 32685 raft_consensus.cc:2454] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 2.
I20240501 14:09:43.099586 29865 catalog_manager.cc:5484] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:43.100776 30344 leader_election.cc:302] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:43.100948 2864 raft_consensus.cc:2781] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:43.101083 2864 raft_consensus.cc:686] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:43.101243 2864 consensus_queue.cc:237] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:43.103307 29865 catalog_manager.cc:5484] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 1 to 2, leader changed from d35c16dcb506426993811bb2519d7db3 (127.22.252.130) to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 2 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:43.123854 29864 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:43.129695 327 tablet_service.cc:1467] Processing CreateTablet for tablet e946b45d487f4c47a0cb67b85b4654ba (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d1390e13e15846da9871d5638da09269]), partition=
I20240501 14:09:43.129976 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e946b45d487f4c47a0cb67b85b4654ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.132092 4140 tablet_bootstrap.cc:492] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:43.132304 29993 tablet_service.cc:1467] Processing CreateTablet for tablet e946b45d487f4c47a0cb67b85b4654ba (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d1390e13e15846da9871d5638da09269]), partition=
I20240501 14:09:43.132546 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e946b45d487f4c47a0cb67b85b4654ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.133051 4140 tablet_bootstrap.cc:654] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.134194 4140 tablet_bootstrap.cc:492] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:43.134269 4140 ts_tablet_manager.cc:1367] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:09:43.134765 4140 raft_consensus.cc:348] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:43.134857 4140 raft_consensus.cc:374] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.134887 4140 raft_consensus.cc:729] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.135171 4142 tablet_bootstrap.cc:492] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:43.135396 32664 tablet_service.cc:1467] Processing CreateTablet for tablet e946b45d487f4c47a0cb67b85b4654ba (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d1390e13e15846da9871d5638da09269]), partition=
I20240501 14:09:43.135649 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e946b45d487f4c47a0cb67b85b4654ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.136674 4142 tablet_bootstrap.cc:654] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.137463 4140 consensus_queue.cc:260] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:43.137938 4140 ts_tablet_manager.cc:1398] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:43.138576 4144 tablet_bootstrap.cc:492] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:43.139415 4142 tablet_bootstrap.cc:492] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:43.139523 4142 ts_tablet_manager.cc:1367] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:43.139546 4144 tablet_bootstrap.cc:654] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.140015 4142 raft_consensus.cc:348] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:43.140136 4142 raft_consensus.cc:374] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.140172 4142 raft_consensus.cc:729] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.140313 4142 consensus_queue.cc:260] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:43.140590 4142 ts_tablet_manager.cc:1398] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.140708 4144 tablet_bootstrap.cc:492] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:43.140782 4144 ts_tablet_manager.cc:1367] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:43.141183 4144 raft_consensus.cc:348] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:43.141285 4144 raft_consensus.cc:374] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.141317 4144 raft_consensus.cc:729] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.141427 4144 consensus_queue.cc:260] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:43.141731 4144 ts_tablet_manager.cc:1398] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.167220 29993 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:43.168536 327 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:43.171183 32534 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:43.175107 32400 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:43.179673 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:43.184221 32664 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:43.193615 3126 raft_consensus.cc:482] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:43.193699 3126 raft_consensus.cc:504] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:43.194036 3126 leader_election.cc:288] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:43.195729 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e946b45d487f4c47a0cb67b85b4654ba" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:43.195860 30013 raft_consensus.cc:2454] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:43.196219 32752 leader_election.cc:302] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:43.196472 3126 raft_consensus.cc:2781] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:43.196523 3126 raft_consensus.cc:482] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:43.196560 3126 raft_consensus.cc:3037] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.196620 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e946b45d487f4c47a0cb67b85b4654ba" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:43.196739 32685 raft_consensus.cc:2454] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:43.197695 3126 raft_consensus.cc:504] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:43.198001 3126 leader_election.cc:288] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:43.198873 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e946b45d487f4c47a0cb67b85b4654ba" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:43.198990 32685 raft_consensus.cc:3037] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.199980 32685 raft_consensus.cc:2454] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:43.200333 32750 leader_election.cc:302] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:43.200537 3126 raft_consensus.cc:2781] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:43.200614 3126 raft_consensus.cc:686] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:43.200735 3126 consensus_queue.cc:237] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:43.200764 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e946b45d487f4c47a0cb67b85b4654ba" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:43.200866 30013 raft_consensus.cc:3037] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.201516 30388 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:43.201818 30013 raft_consensus.cc:2454] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:43.203169 29864 catalog_manager.cc:5484] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:43.205657 30124 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:43.307441 29866 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:43.312137 30013 raft_consensus.cc:1261] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:43.312654 3126 consensus_queue.cc:1035] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:43.313488 32685 raft_consensus.cc:1261] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:43.315296 4152 consensus_queue.cc:1035] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:43.315901 29864 catalog_manager.cc:5576] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387) does not have the latest schema for tablet e946b45d487f4c47a0cb67b85b4654ba (table rebalancer_dynamic_table [id=d1390e13e15846da9871d5638da09269]). Expected version 1 got 0
I20240501 14:09:43.318562 4277 tablet.cc:1722] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:43.319252 4277 tablet.cc:1620] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:43.321282 29866 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet e946b45d487f4c47a0cb67b85b4654ba (table rebalancer_dynamic_table [id=d1390e13e15846da9871d5638da09269]). Expected version 1 got 0
I20240501 14:09:43.321942 29873 catalog_manager.cc:5576] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387) does not have the latest schema for tablet e946b45d487f4c47a0cb67b85b4654ba (table rebalancer_dynamic_table [id=d1390e13e15846da9871d5638da09269]). Expected version 1 got 0
I20240501 14:09:43.325234 4279 tablet.cc:1722] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:43.325639 4279 tablet.cc:1620] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e: MemRowSet was empty: no flush needed.
I20240501 14:09:43.328488 4075 tablet.cc:1722] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:43.328780 4075 tablet.cc:1620] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:43.333803 29873 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=d1390e13e15846da9871d5638da09269] alter complete (version 1)
I20240501 14:09:43.335170 3126 consensus_queue.cc:1035] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:43.341594 4167 consensus_queue.cc:1035] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:43.342804 30388 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:43.344735 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:43.344834 32534 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:43.348050 32400 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:43.356581 327 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:43.364799 29993 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:43.371060 30124 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:43.392751 32664 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:43.434522 30264 consensus_queue.cc:237] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:43.435698 32553 raft_consensus.cc:1261] T 0517ff3519644b489b912f8e5a36274c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:43.436061 30146 raft_consensus.cc:1261] T 0517ff3519644b489b912f8e5a36274c P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:43.436076 3921 consensus_queue.cc:1035] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:43.436473 3921 consensus_queue.cc:1035] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:43.437830 4242 raft_consensus.cc:2932] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:43.438302 32553 raft_consensus.cc:2932] T 0517ff3519644b489b912f8e5a36274c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:43.440093 30146 raft_consensus.cc:2932] T 0517ff3519644b489b912f8e5a36274c P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:43.440815 29864 catalog_manager.cc:5484] T 0517ff3519644b489b912f8e5a36274c P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: config changed from index 4 to 5, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:43.443900 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet 0517ff3519644b489b912f8e5a36274c with delete_type TABLET_DATA_TOMBSTONED (TS 93f2518fbadf4f9991cda93afacb363e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:43.444307 4311 tablet_replica.cc:330] T 0517ff3519644b489b912f8e5a36274c P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:43.444551 4311 raft_consensus.cc:2229] T 0517ff3519644b489b912f8e5a36274c P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:43.444893 4311 raft_consensus.cc:2258] T 0517ff3519644b489b912f8e5a36274c P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:43.445442 4311 ts_tablet_manager.cc:1875] T 0517ff3519644b489b912f8e5a36274c P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:43.446859 29873 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:43.447465 4311 ts_tablet_manager.cc:1888] T 0517ff3519644b489b912f8e5a36274c P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:43.447608 4311 log.cc:1196] T 0517ff3519644b489b912f8e5a36274c P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/0517ff3519644b489b912f8e5a36274c
I20240501 14:09:43.448177 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 0517ff3519644b489b912f8e5a36274c (table rebalance_test_table_3 [id=b7b3f5bc4b6649c5a97ec9f02da4e967]) successfully deleted
I20240501 14:09:43.452145 4277 tablet.cc:1722] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:43.452466 4277 tablet.cc:1620] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:43.454010 4279 tablet.cc:1722] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:43.454277 4279 tablet.cc:1620] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e: MemRowSet was empty: no flush needed.
I20240501 14:09:43.454716 4075 tablet.cc:1722] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:43.455026 4075 tablet.cc:1620] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:43.457690 29873 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=d1390e13e15846da9871d5638da09269] alter complete (version 2)
I20240501 14:09:43.524318 30264 raft_consensus.cc:1261] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:43.524899 2864 consensus_queue.cc:1035] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:43.525218 4167 consensus_queue.cc:1035] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:43.530089 32534 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:43.534201 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:43.538784 32664 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:43.539539 30388 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:43.546514 4167 consensus_queue.cc:1035] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:43.547631 30124 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:43.553437 32685 raft_consensus.cc:1261] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:43.553804 4214 consensus_queue.cc:1035] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:43.557123 4010 consensus_queue.cc:1035] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:43.557363 4010 consensus_queue.cc:1035] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:43.557704 327 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:43.558254 29993 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:43.561491 29864 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:43.568522 4279 tablet.cc:1722] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:43.570340 4075 tablet.cc:1722] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:43.572314 4277 tablet.cc:1722] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:43.573763 30146 raft_consensus.cc:1261] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [term 2 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:43.574153 2864 consensus_queue.cc:1035] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:43.576292 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=d1390e13e15846da9871d5638da09269] alter complete (version 3)
I20240501 14:09:43.612116 29866 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:43.612713 32400 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:43.618536 32534 tablet_service.cc:1467] Processing CreateTablet for tablet bd36b13ca2274fc19b9d32d202e94f83 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=1e152e3403f94ab998a842bc24585ae1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.618837 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd36b13ca2274fc19b9d32d202e94f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.621577 327 tablet_service.cc:1467] Processing CreateTablet for tablet bd36b13ca2274fc19b9d32d202e94f83 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=1e152e3403f94ab998a842bc24585ae1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.621851 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd36b13ca2274fc19b9d32d202e94f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.621842 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 3110aca9b1d643e4b07d0a869015b6f7 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=1e152e3403f94ab998a842bc24585ae1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.622058 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3110aca9b1d643e4b07d0a869015b6f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.623873 4142 tablet_bootstrap.cc:492] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:43.623960 4140 tablet_bootstrap.cc:492] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:43.624632 32402 tablet_service.cc:1467] Processing CreateTablet for tablet ce04f87c84f54f1aaabb7cf10afe5b50 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=1e152e3403f94ab998a842bc24585ae1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.624891 32402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce04f87c84f54f1aaabb7cf10afe5b50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.624935 4140 tablet_bootstrap.cc:654] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.624950 4142 tablet_bootstrap.cc:654] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.625748 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 3110aca9b1d643e4b07d0a869015b6f7 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=1e152e3403f94ab998a842bc24585ae1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.625962 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3110aca9b1d643e4b07d0a869015b6f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.626014 4142 tablet_bootstrap.cc:492] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:43.626091 4142 ts_tablet_manager.cc:1367] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:43.626511 4142 raft_consensus.cc:348] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:43.626610 4142 raft_consensus.cc:374] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.626650 4142 raft_consensus.cc:729] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.626776 4142 consensus_queue.cc:260] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:43.627070 4142 ts_tablet_manager.cc:1398] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.628082 4377 tablet_bootstrap.cc:492] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:43.628494 29995 tablet_service.cc:1467] Processing CreateTablet for tablet ce04f87c84f54f1aaabb7cf10afe5b50 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=1e152e3403f94ab998a842bc24585ae1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.628727 29995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce04f87c84f54f1aaabb7cf10afe5b50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.629621 4377 tablet_bootstrap.cc:654] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.630364 4378 tablet_bootstrap.cc:492] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:43.630738 4142 tablet_bootstrap.cc:492] T ce04f87c84f54f1aaabb7cf10afe5b50 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:43.630889 4377 tablet_bootstrap.cc:492] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:43.630964 4377 ts_tablet_manager.cc:1367] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:43.630991 30124 tablet_service.cc:1467] Processing CreateTablet for tablet bd36b13ca2274fc19b9d32d202e94f83 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=1e152e3403f94ab998a842bc24585ae1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.631228 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd36b13ca2274fc19b9d32d202e94f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.631409 4377 raft_consensus.cc:348] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:43.631539 4377 raft_consensus.cc:374] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.631556 4142 tablet_bootstrap.cc:654] T ce04f87c84f54f1aaabb7cf10afe5b50 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.631570 4377 raft_consensus.cc:729] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.631671 4377 consensus_queue.cc:260] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:43.631903 4377 ts_tablet_manager.cc:1398] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:43.632670 4378 tablet_bootstrap.cc:654] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.632668 4377 tablet_bootstrap.cc:492] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:43.633513 4377 tablet_bootstrap.cc:654] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.633666 4142 tablet_bootstrap.cc:492] T ce04f87c84f54f1aaabb7cf10afe5b50 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:43.633733 4142 ts_tablet_manager.cc:1367] T ce04f87c84f54f1aaabb7cf10afe5b50 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:43.634191 4142 raft_consensus.cc:348] T ce04f87c84f54f1aaabb7cf10afe5b50 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:43.634282 4142 raft_consensus.cc:374] T ce04f87c84f54f1aaabb7cf10afe5b50 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.634310 4142 raft_consensus.cc:729] T ce04f87c84f54f1aaabb7cf10afe5b50 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.634409 4142 consensus_queue.cc:260] T ce04f87c84f54f1aaabb7cf10afe5b50 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:43.634634 4142 ts_tablet_manager.cc:1398] T ce04f87c84f54f1aaabb7cf10afe5b50 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.635481 32532 tablet_service.cc:1467] Processing CreateTablet for tablet 3110aca9b1d643e4b07d0a869015b6f7 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=1e152e3403f94ab998a842bc24585ae1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.635713 32532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3110aca9b1d643e4b07d0a869015b6f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.635991 4378 tablet_bootstrap.cc:492] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:43.636075 4378 ts_tablet_manager.cc:1367] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:43.636683 4378 raft_consensus.cc:348] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:43.636786 4378 raft_consensus.cc:374] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.636826 4378 raft_consensus.cc:729] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.636953 4378 consensus_queue.cc:260] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:43.637346 4378 ts_tablet_manager.cc:1398] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.637845 4377 tablet_bootstrap.cc:492] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:43.637874 4378 tablet_bootstrap.cc:492] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:43.637931 4377 ts_tablet_manager.cc:1367] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:43.638370 4377 raft_consensus.cc:348] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:43.638506 4377 raft_consensus.cc:374] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.638556 4377 raft_consensus.cc:729] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.638674 4377 consensus_queue.cc:260] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:43.638950 4377 ts_tablet_manager.cc:1398] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.639396 30256 tablet_service.cc:1467] Processing CreateTablet for tablet ce04f87c84f54f1aaabb7cf10afe5b50 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=1e152e3403f94ab998a842bc24585ae1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.639712 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce04f87c84f54f1aaabb7cf10afe5b50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.642046 4383 tablet_bootstrap.cc:492] T ce04f87c84f54f1aaabb7cf10afe5b50 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:43.642360 4378 tablet_bootstrap.cc:654] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.642904 4383 tablet_bootstrap.cc:654] T ce04f87c84f54f1aaabb7cf10afe5b50 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.643322 4378 tablet_bootstrap.cc:492] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:43.643395 4378 ts_tablet_manager.cc:1367] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:43.643762 4378 raft_consensus.cc:348] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:43.643846 4378 raft_consensus.cc:374] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.643872 4378 raft_consensus.cc:729] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.643980 4378 consensus_queue.cc:260] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:43.644207 4378 ts_tablet_manager.cc:1398] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:43.647089 4140 tablet_bootstrap.cc:492] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:43.647189 4140 ts_tablet_manager.cc:1367] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240501 14:09:43.647606 4140 raft_consensus.cc:348] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:43.647712 4140 raft_consensus.cc:374] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.647742 4140 raft_consensus.cc:729] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.647850 4140 consensus_queue.cc:260] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:43.648101 4140 ts_tablet_manager.cc:1398] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.655289 4386 tablet_bootstrap.cc:492] T bd36b13ca2274fc19b9d32d202e94f83 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:43.656181 4386 tablet_bootstrap.cc:654] T bd36b13ca2274fc19b9d32d202e94f83 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.664847 4383 tablet_bootstrap.cc:492] T ce04f87c84f54f1aaabb7cf10afe5b50 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:43.664943 4383 ts_tablet_manager.cc:1367] T ce04f87c84f54f1aaabb7cf10afe5b50 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.000s
I20240501 14:09:43.666610 4383 raft_consensus.cc:348] T ce04f87c84f54f1aaabb7cf10afe5b50 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:43.667116 4383 raft_consensus.cc:374] T ce04f87c84f54f1aaabb7cf10afe5b50 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.667217 4383 raft_consensus.cc:729] T ce04f87c84f54f1aaabb7cf10afe5b50 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.667455 4383 consensus_queue.cc:260] T ce04f87c84f54f1aaabb7cf10afe5b50 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:43.667831 4383 ts_tablet_manager.cc:1398] T ce04f87c84f54f1aaabb7cf10afe5b50 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:43.668180 4386 tablet_bootstrap.cc:492] T bd36b13ca2274fc19b9d32d202e94f83 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:43.668354 4386 ts_tablet_manager.cc:1367] T bd36b13ca2274fc19b9d32d202e94f83 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:09:43.668855 4386 raft_consensus.cc:348] T bd36b13ca2274fc19b9d32d202e94f83 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:43.669068 4386 raft_consensus.cc:374] T bd36b13ca2274fc19b9d32d202e94f83 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.669117 4386 raft_consensus.cc:729] T bd36b13ca2274fc19b9d32d202e94f83 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.669273 4386 consensus_queue.cc:260] T bd36b13ca2274fc19b9d32d202e94f83 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:43.669615 4386 ts_tablet_manager.cc:1398] T bd36b13ca2274fc19b9d32d202e94f83 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.681677 29864 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:43.681810 29864 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:43.684034 29864 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet e946b45d487f4c47a0cb67b85b4654ba
I20240501 14:09:43.684679 29995 tablet_service.cc:1514] Processing DeleteTablet for tablet e946b45d487f4c47a0cb67b85b4654ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:43 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:43.684767 327 tablet_service.cc:1514] Processing DeleteTablet for tablet e946b45d487f4c47a0cb67b85b4654ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:43 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:43.684764 32664 tablet_service.cc:1514] Processing DeleteTablet for tablet e946b45d487f4c47a0cb67b85b4654ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:43 UTC) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:43.684911 4311 tablet_replica.cc:330] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:43.685051 4311 raft_consensus.cc:2229] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:43.685070 4399 tablet_replica.cc:330] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:43.685181 4311 raft_consensus.cc:2258] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:43.685226 4399 raft_consensus.cc:2229] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:43.685436 4398 tablet_replica.cc:330] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:43.685505 4399 raft_consensus.cc:2258] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:43.685585 4398 raft_consensus.cc:2229] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:43.685804 4398 raft_consensus.cc:2258] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:43.686136 4399 ts_tablet_manager.cc:1875] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:43.686395 4398 ts_tablet_manager.cc:1875] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:43.686558 4311 ts_tablet_manager.cc:1875] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:43.688830 4399 ts_tablet_manager.cc:1888] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:43.688954 4399 log.cc:1196] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/e946b45d487f4c47a0cb67b85b4654ba
I20240501 14:09:43.689011 4311 ts_tablet_manager.cc:1888] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:43.689163 4311 log.cc:1196] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/e946b45d487f4c47a0cb67b85b4654ba
I20240501 14:09:43.689308 4399 ts_tablet_manager.cc:1909] T e946b45d487f4c47a0cb67b85b4654ba P db935501e9104de5b15c8780681d6c38: Deleting consensus metadata
I20240501 14:09:43.689500 4311 ts_tablet_manager.cc:1909] T e946b45d487f4c47a0cb67b85b4654ba P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:43.689963 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet e946b45d487f4c47a0cb67b85b4654ba (table rebalancer_dynamic_table_new_name [id=d1390e13e15846da9871d5638da09269]) successfully deleted
I20240501 14:09:43.690143 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet e946b45d487f4c47a0cb67b85b4654ba (table rebalancer_dynamic_table_new_name [id=d1390e13e15846da9871d5638da09269]) successfully deleted
I20240501 14:09:43.690894 4398 ts_tablet_manager.cc:1888] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:43.691036 4398 log.cc:1196] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/e946b45d487f4c47a0cb67b85b4654ba
I20240501 14:09:43.691284 4398 ts_tablet_manager.cc:1909] T e946b45d487f4c47a0cb67b85b4654ba P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:09:43.691885 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet e946b45d487f4c47a0cb67b85b4654ba (table rebalancer_dynamic_table_new_name [id=d1390e13e15846da9871d5638da09269]) successfully deleted
I20240501 14:09:43.695912 348 consensus_queue.cc:237] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:43.696985 30013 raft_consensus.cc:1261] T 66471362a1d24a8dab53f9cad512a73a P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:43.697329 30406 raft_consensus.cc:1261] T 66471362a1d24a8dab53f9cad512a73a P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:43.697342 3126 consensus_queue.cc:1035] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:43.697688 3126 consensus_queue.cc:1035] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:43.699393 4152 raft_consensus.cc:2932] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:43.699805 30013 raft_consensus.cc:2932] T 66471362a1d24a8dab53f9cad512a73a P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:43.699911 30406 raft_consensus.cc:2932] T 66471362a1d24a8dab53f9cad512a73a P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:43.702329 29864 catalog_manager.cc:5484] T 66471362a1d24a8dab53f9cad512a73a P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:43.704985 32400 tablet_service.cc:1514] Processing DeleteTablet for tablet 66471362a1d24a8dab53f9cad512a73a with delete_type TABLET_DATA_TOMBSTONED (TS a91029b90bf9415a89a2f70736759ad3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:43.706007 4408 tablet_replica.cc:330] T 66471362a1d24a8dab53f9cad512a73a P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:43.706172 4408 raft_consensus.cc:2229] T 66471362a1d24a8dab53f9cad512a73a P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:43.706344 4408 raft_consensus.cc:2258] T 66471362a1d24a8dab53f9cad512a73a P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:43.706825 4408 ts_tablet_manager.cc:1875] T 66471362a1d24a8dab53f9cad512a73a P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:43.709048 4408 ts_tablet_manager.cc:1888] T 66471362a1d24a8dab53f9cad512a73a P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:09:43.709107 4408 log.cc:1196] T 66471362a1d24a8dab53f9cad512a73a P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/66471362a1d24a8dab53f9cad512a73a
I20240501 14:09:43.709635 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 66471362a1d24a8dab53f9cad512a73a (table rebalancer_extra_table_2 [id=e53010e4736c42a899edac137f9182cb]) successfully deleted
I20240501 14:09:43.763605 3954 raft_consensus.cc:482] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:43.763728 3954 raft_consensus.cc:504] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:43.764101 3954 leader_election.cc:288] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:43.764554 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3110aca9b1d643e4b07d0a869015b6f7" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:43.764690 30013 raft_consensus.cc:2454] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:43.764711 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3110aca9b1d643e4b07d0a869015b6f7" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:43.764865 32416 raft_consensus.cc:2454] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:43.764937 32489 leader_election.cc:302] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:43.765198 3954 raft_consensus.cc:2781] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:43.765254 3954 raft_consensus.cc:482] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:43.765301 3954 raft_consensus.cc:3037] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.766440 3954 raft_consensus.cc:504] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:43.766748 3954 leader_election.cc:288] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:43.767206 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3110aca9b1d643e4b07d0a869015b6f7" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:43.767328 30013 raft_consensus.cc:3037] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.767304 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3110aca9b1d643e4b07d0a869015b6f7" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:43.767395 32416 raft_consensus.cc:3037] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.768294 32416 raft_consensus.cc:2454] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:43.768294 30013 raft_consensus.cc:2454] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:43.768865 32489 leader_election.cc:302] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:43.769127 3954 raft_consensus.cc:2781] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:43.769272 3954 raft_consensus.cc:686] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:09:43.769412 3954 consensus_queue.cc:237] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:43.771662 29868 catalog_manager.cc:5484] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:43.785774 29864 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:43.790922 32532 tablet_service.cc:1467] Processing CreateTablet for tablet 558c5e9e95ff4086acc1bde63a4c7e8d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=02ebac92b4574702a256ff45701b948d]), partition=
I20240501 14:09:43.791174 32532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558c5e9e95ff4086acc1bde63a4c7e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.793308 4378 tablet_bootstrap.cc:492] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:43.794258 4378 tablet_bootstrap.cc:654] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.794283 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 558c5e9e95ff4086acc1bde63a4c7e8d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=02ebac92b4574702a256ff45701b948d]), partition=
I20240501 14:09:43.794507 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558c5e9e95ff4086acc1bde63a4c7e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.795094 327 tablet_service.cc:1467] Processing CreateTablet for tablet 558c5e9e95ff4086acc1bde63a4c7e8d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=02ebac92b4574702a256ff45701b948d]), partition=
I20240501 14:09:43.795307 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558c5e9e95ff4086acc1bde63a4c7e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.797613 4377 tablet_bootstrap.cc:492] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:43.798480 4377 tablet_bootstrap.cc:654] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.799135 4140 tablet_bootstrap.cc:492] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:43.799959 4140 tablet_bootstrap.cc:654] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.801122 4377 tablet_bootstrap.cc:492] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:43.801199 4377 ts_tablet_manager.cc:1367] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:43.801203 4140 tablet_bootstrap.cc:492] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:43.801275 4140 ts_tablet_manager.cc:1367] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:43.801724 4140 raft_consensus.cc:348] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:43.801832 4140 raft_consensus.cc:374] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.801769 4377 raft_consensus.cc:348] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:43.801872 4377 raft_consensus.cc:374] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.801867 4140 raft_consensus.cc:729] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.801900 4377 raft_consensus.cc:729] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.802006 4377 consensus_queue.cc:260] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:43.802002 4140 consensus_queue.cc:260] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:43.802253 4377 ts_tablet_manager.cc:1398] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.802271 4140 ts_tablet_manager.cc:1398] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:43.805006 4450 raft_consensus.cc:482] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:43.805290 4450 raft_consensus.cc:504] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:43.805877 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd36b13ca2274fc19b9d32d202e94f83" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:43.805999 348 raft_consensus.cc:2454] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:43.806221 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd36b13ca2274fc19b9d32d202e94f83" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:43.806350 30146 raft_consensus.cc:2454] T bd36b13ca2274fc19b9d32d202e94f83 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:43.806699 32490 leader_election.cc:302] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:43.806844 3954 raft_consensus.cc:2781] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:43.806903 3954 raft_consensus.cc:482] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:43.806952 3954 raft_consensus.cc:3037] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.807324 4450 leader_election.cc:288] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:43.807889 3954 raft_consensus.cc:504] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:43.808193 3954 leader_election.cc:288] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:43.808456 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd36b13ca2274fc19b9d32d202e94f83" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:43.808547 348 raft_consensus.cc:3037] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.809234 348 raft_consensus.cc:2454] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:43.809489 32489 leader_election.cc:302] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:43.809569 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd36b13ca2274fc19b9d32d202e94f83" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:43.809664 30146 raft_consensus.cc:3037] T bd36b13ca2274fc19b9d32d202e94f83 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.810635 30146 raft_consensus.cc:2454] T bd36b13ca2274fc19b9d32d202e94f83 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:43.811303 3954 raft_consensus.cc:2781] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:43.811386 3954 raft_consensus.cc:686] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:09:43.811511 3954 consensus_queue.cc:237] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:43.811722 4378 tablet_bootstrap.cc:492] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:43.811793 4378 ts_tablet_manager.cc:1367] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 14:09:43.812225 4378 raft_consensus.cc:348] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:43.812333 4378 raft_consensus.cc:374] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.812362 4378 raft_consensus.cc:729] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.812459 4378 consensus_queue.cc:260] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:43.812736 4378 ts_tablet_manager.cc:1398] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.813551 29872 catalog_manager.cc:5484] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:43.824306 2503 raft_consensus.cc:482] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:43.824398 2503 raft_consensus.cc:504] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:43.824733 2503 leader_election.cc:288] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:43.825119 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce04f87c84f54f1aaabb7cf10afe5b50" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:43.825122 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce04f87c84f54f1aaabb7cf10afe5b50" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:43.825256 30264 raft_consensus.cc:2454] T ce04f87c84f54f1aaabb7cf10afe5b50 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:43.825256 30013 raft_consensus.cc:2454] T ce04f87c84f54f1aaabb7cf10afe5b50 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:43.825520 32358 leader_election.cc:302] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:43.825693 2503 raft_consensus.cc:2781] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:43.825748 2503 raft_consensus.cc:482] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:43.825801 2503 raft_consensus.cc:3037] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.826571 2503 raft_consensus.cc:504] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:43.826889 2503 leader_election.cc:288] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:43.827239 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce04f87c84f54f1aaabb7cf10afe5b50" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:43.827333 30264 raft_consensus.cc:3037] T ce04f87c84f54f1aaabb7cf10afe5b50 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.827356 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce04f87c84f54f1aaabb7cf10afe5b50" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:43.827462 30013 raft_consensus.cc:3037] T ce04f87c84f54f1aaabb7cf10afe5b50 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.828281 30264 raft_consensus.cc:2454] T ce04f87c84f54f1aaabb7cf10afe5b50 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:43.828430 30013 raft_consensus.cc:2454] T ce04f87c84f54f1aaabb7cf10afe5b50 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:43.828994 32358 leader_election.cc:302] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:43.829321 2503 raft_consensus.cc:2781] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:43.829397 2503 raft_consensus.cc:686] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:43.829525 2503 consensus_queue.cc:237] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:43.831405 29872 catalog_manager.cc:5484] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:43.854849 29995 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:43.855671 32532 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:43.865510 30388 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:43.872586 32664 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:43.872874 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:43.882213 327 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:43.882367 32400 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:43.887173 4450 raft_consensus.cc:482] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:43.887286 4450 raft_consensus.cc:504] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:43.887671 4450 leader_election.cc:288] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:43.888127 30124 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:43.888196 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558c5e9e95ff4086acc1bde63a4c7e8d" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:43.888330 348 raft_consensus.cc:2454] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:43.888633 32489 leader_election.cc:302] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:43.888667 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558c5e9e95ff4086acc1bde63a4c7e8d" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:43.888779 32416 raft_consensus.cc:2454] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:43.888821 4450 raft_consensus.cc:2781] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:43.888862 4450 raft_consensus.cc:482] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:43.888919 4450 raft_consensus.cc:3037] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.889994 4450 raft_consensus.cc:504] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:43.890326 4450 leader_election.cc:288] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:43.903699 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558c5e9e95ff4086acc1bde63a4c7e8d" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:43.903703 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558c5e9e95ff4086acc1bde63a4c7e8d" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:43.903811 32416 raft_consensus.cc:3037] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.903812 348 raft_consensus.cc:3037] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.904798 348 raft_consensus.cc:2454] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:43.904800 32416 raft_consensus.cc:2454] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:43.905282 32489 leader_election.cc:302] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:43.905640 3954 raft_consensus.cc:2781] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:43.905715 3954 raft_consensus.cc:686] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:09:43.905839 3954 consensus_queue.cc:237] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:43.907833 29872 catalog_manager.cc:5484] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:43.943959 30264 consensus_queue.cc:237] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:43.945060 32416 raft_consensus.cc:1261] T 4a5869fb581b4939ad2207ab9200d674 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:43.945160 30013 raft_consensus.cc:1261] T 4a5869fb581b4939ad2207ab9200d674 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:43.945477 3921 consensus_queue.cc:1035] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:43.945643 4242 consensus_queue.cc:1035] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:43.947489 4488 raft_consensus.cc:2932] T 4a5869fb581b4939ad2207ab9200d674 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:43.947832 32416 raft_consensus.cc:2932] T 4a5869fb581b4939ad2207ab9200d674 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:43.947906 30013 raft_consensus.cc:2932] T 4a5869fb581b4939ad2207ab9200d674 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:43.950860 29872 catalog_manager.cc:5484] T 4a5869fb581b4939ad2207ab9200d674 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:43.954834 30124 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a5869fb581b4939ad2207ab9200d674 with delete_type TABLET_DATA_TOMBSTONED (TS d35c16dcb506426993811bb2519d7db3 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:43.955190 4489 tablet_replica.cc:330] T 4a5869fb581b4939ad2207ab9200d674 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:43.955339 4489 raft_consensus.cc:2229] T 4a5869fb581b4939ad2207ab9200d674 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:43.955492 4489 raft_consensus.cc:2258] T 4a5869fb581b4939ad2207ab9200d674 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:43.956276 4489 ts_tablet_manager.cc:1875] T 4a5869fb581b4939ad2207ab9200d674 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:43.958438 4489 ts_tablet_manager.cc:1888] T 4a5869fb581b4939ad2207ab9200d674 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:43.958549 4489 log.cc:1196] T 4a5869fb581b4939ad2207ab9200d674 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/4a5869fb581b4939ad2207ab9200d674
I20240501 14:09:43.959481 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 4a5869fb581b4939ad2207ab9200d674 (table rebalance_test_table_17 [id=9a4b9a0a2fe64ab2a64d6679185c2a33]) successfully deleted
I20240501 14:09:44.026233 29870 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:44.032003 32416 raft_consensus.cc:1261] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:44.032038 348 raft_consensus.cc:1261] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:44.032634 4468 consensus_queue.cc:1035] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:44.032910 4450 consensus_queue.cc:1035] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:44.036429 29867 catalog_manager.cc:5576] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265) does not have the latest schema for tablet 558c5e9e95ff4086acc1bde63a4c7e8d (table rebalancer_dynamic_table [id=02ebac92b4574702a256ff45701b948d]). Expected version 1 got 0
I20240501 14:09:44.036655 4277 tablet.cc:1722] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:44.036943 3767 mvcc.cc:205] Tried to move back new op lower bound from 7022889304191049728 to 7022889303679680512. Current Snapshot: MvccSnapshot[applied={T|T < 7022889304191049728}]
I20240501 14:09:44.037015 4277 tablet.cc:1620] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:44.038092 29872 catalog_manager.cc:5576] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937) does not have the latest schema for tablet 558c5e9e95ff4086acc1bde63a4c7e8d (table rebalancer_dynamic_table [id=02ebac92b4574702a256ff45701b948d]). Expected version 1 got 0
I20240501 14:09:44.038798 4534 tablet.cc:1722] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:44.039191 4534 tablet.cc:1620] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:44.040182 4535 tablet.cc:1722] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:44.041313 4535 tablet.cc:1620] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:44.043267 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=02ebac92b4574702a256ff45701b948d] alter complete (version 1)
I20240501 14:09:44.043509 29870 catalog_manager.cc:5576] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265) does not have the latest schema for tablet 558c5e9e95ff4086acc1bde63a4c7e8d (table rebalancer_dynamic_table [id=02ebac92b4574702a256ff45701b948d]). Expected version 1 got 0
I20240501 14:09:44.060354 327 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:44.060760 32532 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:44.062264 32400 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:44.063592 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:44.072422 32664 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:44.073926 29995 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:44.083526 30124 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:44.097258 30388 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:44.138089 30264 consensus_queue.cc:237] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:44.139636 32553 raft_consensus.cc:1261] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.139830 30146 raft_consensus.cc:1261] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.140184 4242 consensus_queue.cc:1035] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.140403 3921 consensus_queue.cc:1035] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.142127 4242 raft_consensus.cc:2932] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:44.142599 32553 raft_consensus.cc:2932] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:44.142642 30146 raft_consensus.cc:2932] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:44.145460 29870 catalog_manager.cc:5484] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d35c16dcb506426993811bb2519d7db3 reported cstate change: config changed from index 4 to 5, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:44.148846 30388 tablet_service.cc:1514] Processing DeleteTablet for tablet 44949ddf1eac45cfa4fc8e1c532cf7bb with delete_type TABLET_DATA_TOMBSTONED (TS d1c217469ae4478ebd49e4ed9f7352d7 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:44.149163 4565 tablet_replica.cc:330] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:44.149524 4565 raft_consensus.cc:2229] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.149708 4565 raft_consensus.cc:2258] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.150180 4565 ts_tablet_manager.cc:1875] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:44.152361 29870 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:44.153556 4565 ts_tablet_manager.cc:1888] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:44.153700 4565 log.cc:1196] T 44949ddf1eac45cfa4fc8e1c532cf7bb P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/44949ddf1eac45cfa4fc8e1c532cf7bb
I20240501 14:09:44.154311 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 44949ddf1eac45cfa4fc8e1c532cf7bb (table rebalance_test_table_17 [id=9a4b9a0a2fe64ab2a64d6679185c2a33]) successfully deleted
I20240501 14:09:44.157939 4534 tablet.cc:1722] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:44.158221 4534 tablet.cc:1620] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:44.158912 4535 tablet.cc:1722] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:44.159229 4535 tablet.cc:1620] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:44.161849 4277 tablet.cc:1722] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:44.162174 4277 tablet.cc:1620] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:44.165755 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=02ebac92b4574702a256ff45701b948d] alter complete (version 2)
I20240501 14:09:44.213697 4010 consensus_queue.cc:1035] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:44.216547 4010 consensus_queue.cc:1035] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:44.233923 4450 consensus_queue.cc:1035] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:44.240535 4468 consensus_queue.cc:1035] T bd36b13ca2274fc19b9d32d202e94f83 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:44.262075 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:44.264222 327 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:44.267541 32532 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:44.267762 29870 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:44.268005 32400 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:44.275084 30124 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:44.276041 4534 tablet.cc:1722] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:44.276846 4277 tablet.cc:1722] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:44.279227 4535 tablet.cc:1722] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:44.281476 29866 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=02ebac92b4574702a256ff45701b948d] alter complete (version 3)
I20240501 14:09:44.292485 30388 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:44.292496 32664 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:44.306094 29995 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:44.319391 4468 consensus_queue.cc:1035] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:44.331511 4468 consensus_queue.cc:1035] T 3110aca9b1d643e4b07d0a869015b6f7 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:44.337841 29870 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:44.344494 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 1a277d983e2a47fda9410e1fb8c31221 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=7bd8d12b69014e03985940347a68e560]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.344471 30124 tablet_service.cc:1467] Processing CreateTablet for tablet 1a277d983e2a47fda9410e1fb8c31221 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=7bd8d12b69014e03985940347a68e560]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.344760 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a277d983e2a47fda9410e1fb8c31221. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.344761 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a277d983e2a47fda9410e1fb8c31221. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.345209 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 1a277d983e2a47fda9410e1fb8c31221 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=7bd8d12b69014e03985940347a68e560]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.345484 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a277d983e2a47fda9410e1fb8c31221. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.347538 32532 tablet_service.cc:1467] Processing CreateTablet for tablet 50dc380460bb4f859d25ec77291faffb (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=7bd8d12b69014e03985940347a68e560]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.347813 32532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50dc380460bb4f859d25ec77291faffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.347827 4639 tablet_bootstrap.cc:492] T 1a277d983e2a47fda9410e1fb8c31221 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:44.347868 327 tablet_service.cc:1467] Processing CreateTablet for tablet 50dc380460bb4f859d25ec77291faffb (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=7bd8d12b69014e03985940347a68e560]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.348157 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50dc380460bb4f859d25ec77291faffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.348176 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 50dc380460bb4f859d25ec77291faffb (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=7bd8d12b69014e03985940347a68e560]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.348443 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50dc380460bb4f859d25ec77291faffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.349792 4641 tablet_bootstrap.cc:492] T 50dc380460bb4f859d25ec77291faffb P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:44.349977 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 81750b74ac3f48e082e6e6d62ef8f066 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=7bd8d12b69014e03985940347a68e560]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.350251 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81750b74ac3f48e082e6e6d62ef8f066. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.350816 30256 tablet_service.cc:1467] Processing CreateTablet for tablet 81750b74ac3f48e082e6e6d62ef8f066 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=7bd8d12b69014e03985940347a68e560]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.351095 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81750b74ac3f48e082e6e6d62ef8f066. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.351661 4639 tablet_bootstrap.cc:654] T 1a277d983e2a47fda9410e1fb8c31221 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.352355 4638 tablet_bootstrap.cc:492] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:44.352955 4640 tablet_bootstrap.cc:492] T 1a277d983e2a47fda9410e1fb8c31221 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:44.353106 4644 tablet_bootstrap.cc:492] T 81750b74ac3f48e082e6e6d62ef8f066 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:44.353133 4638 tablet_bootstrap.cc:654] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.353729 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 81750b74ac3f48e082e6e6d62ef8f066 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=7bd8d12b69014e03985940347a68e560]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.353793 4642 tablet_bootstrap.cc:492] T 50dc380460bb4f859d25ec77291faffb P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:44.353936 4640 tablet_bootstrap.cc:654] T 1a277d983e2a47fda9410e1fb8c31221 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.353986 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81750b74ac3f48e082e6e6d62ef8f066. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.354457 4644 tablet_bootstrap.cc:654] T 81750b74ac3f48e082e6e6d62ef8f066 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.354719 4642 tablet_bootstrap.cc:654] T 50dc380460bb4f859d25ec77291faffb P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.354962 4640 tablet_bootstrap.cc:492] T 1a277d983e2a47fda9410e1fb8c31221 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:44.355042 4640 ts_tablet_manager.cc:1367] T 1a277d983e2a47fda9410e1fb8c31221 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:44.355530 4644 tablet_bootstrap.cc:492] T 81750b74ac3f48e082e6e6d62ef8f066 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:44.355475 4640 raft_consensus.cc:348] T 1a277d983e2a47fda9410e1fb8c31221 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:44.355585 4640 raft_consensus.cc:374] T 1a277d983e2a47fda9410e1fb8c31221 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.355589 4639 tablet_bootstrap.cc:492] T 1a277d983e2a47fda9410e1fb8c31221 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:44.355608 4644 ts_tablet_manager.cc:1367] T 81750b74ac3f48e082e6e6d62ef8f066 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:44.355623 4640 raft_consensus.cc:729] T 1a277d983e2a47fda9410e1fb8c31221 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.355662 4639 ts_tablet_manager.cc:1367] T 1a277d983e2a47fda9410e1fb8c31221 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:44.355734 4640 consensus_queue.cc:260] T 1a277d983e2a47fda9410e1fb8c31221 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:44.356026 4640 ts_tablet_manager.cc:1398] T 1a277d983e2a47fda9410e1fb8c31221 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:44.356048 4644 raft_consensus.cc:348] T 81750b74ac3f48e082e6e6d62ef8f066 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:44.356056 4639 raft_consensus.cc:348] T 1a277d983e2a47fda9410e1fb8c31221 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:44.356141 4639 raft_consensus.cc:374] T 1a277d983e2a47fda9410e1fb8c31221 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.356141 4644 raft_consensus.cc:374] T 81750b74ac3f48e082e6e6d62ef8f066 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.356173 4639 raft_consensus.cc:729] T 1a277d983e2a47fda9410e1fb8c31221 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.356173 4644 raft_consensus.cc:729] T 81750b74ac3f48e082e6e6d62ef8f066 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.356197 4640 tablet_bootstrap.cc:492] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:44.356313 4644 consensus_queue.cc:260] T 81750b74ac3f48e082e6e6d62ef8f066 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:44.356452 4639 consensus_queue.cc:260] T 1a277d983e2a47fda9410e1fb8c31221 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:44.356580 4644 ts_tablet_manager.cc:1398] T 81750b74ac3f48e082e6e6d62ef8f066 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:44.356740 4639 ts_tablet_manager.cc:1398] T 1a277d983e2a47fda9410e1fb8c31221 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:44.357177 4640 tablet_bootstrap.cc:654] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.357374 4638 tablet_bootstrap.cc:492] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:44.357445 4638 ts_tablet_manager.cc:1367] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:44.357820 4638 raft_consensus.cc:348] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:44.357920 4638 raft_consensus.cc:374] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.357954 4638 raft_consensus.cc:729] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.358093 4641 tablet_bootstrap.cc:654] T 50dc380460bb4f859d25ec77291faffb P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.358062 4638 consensus_queue.cc:260] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:44.358453 4638 ts_tablet_manager.cc:1398] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:44.358592 4638 tablet_bootstrap.cc:492] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:44.358971 4645 tablet_bootstrap.cc:492] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:44.359429 4638 tablet_bootstrap.cc:654] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.361119 4642 tablet_bootstrap.cc:492] T 50dc380460bb4f859d25ec77291faffb P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:44.361275 4642 ts_tablet_manager.cc:1367] T 50dc380460bb4f859d25ec77291faffb P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:44.361785 4642 raft_consensus.cc:348] T 50dc380460bb4f859d25ec77291faffb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:44.361951 4642 raft_consensus.cc:374] T 50dc380460bb4f859d25ec77291faffb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.362037 4642 raft_consensus.cc:729] T 50dc380460bb4f859d25ec77291faffb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.362210 4642 consensus_queue.cc:260] T 50dc380460bb4f859d25ec77291faffb P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:44.362383 4645 tablet_bootstrap.cc:654] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.362638 4642 ts_tablet_manager.cc:1398] T 50dc380460bb4f859d25ec77291faffb P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:44.363252 4640 tablet_bootstrap.cc:492] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:44.363334 4640 ts_tablet_manager.cc:1367] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:44.363641 4641 tablet_bootstrap.cc:492] T 50dc380460bb4f859d25ec77291faffb P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:44.363768 4641 ts_tablet_manager.cc:1367] T 50dc380460bb4f859d25ec77291faffb P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:09:44.364115 4638 tablet_bootstrap.cc:492] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:44.364284 4638 ts_tablet_manager.cc:1367] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:44.364718 4638 raft_consensus.cc:348] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:44.364800 4638 raft_consensus.cc:374] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.364833 4638 raft_consensus.cc:729] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.364943 4640 raft_consensus.cc:348] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:44.365087 4640 raft_consensus.cc:374] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.365135 4640 raft_consensus.cc:729] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.365312 4640 consensus_queue.cc:260] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:44.365415 4641 raft_consensus.cc:348] T 50dc380460bb4f859d25ec77291faffb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:44.365523 4641 raft_consensus.cc:374] T 50dc380460bb4f859d25ec77291faffb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.365558 4641 raft_consensus.cc:729] T 50dc380460bb4f859d25ec77291faffb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.365681 4641 consensus_queue.cc:260] T 50dc380460bb4f859d25ec77291faffb P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:44.365016 4638 consensus_queue.cc:260] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:44.366226 4638 ts_tablet_manager.cc:1398] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:44.367391 4641 ts_tablet_manager.cc:1398] T 50dc380460bb4f859d25ec77291faffb P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:44.368094 4645 tablet_bootstrap.cc:492] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:44.368175 4645 ts_tablet_manager.cc:1367] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:44.368638 4645 raft_consensus.cc:348] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:44.368743 4645 raft_consensus.cc:374] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.368805 4645 raft_consensus.cc:729] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.368979 4645 consensus_queue.cc:260] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:44.369323 4645 ts_tablet_manager.cc:1398] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:44.372148 4640 ts_tablet_manager.cc:1398] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240501 14:09:44.373158 32416 consensus_queue.cc:237] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:44.374233 30264 raft_consensus.cc:1261] T 588dd26e1820409a9dfccd8ea6a30f49 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:44.374409 30406 raft_consensus.cc:1261] T 588dd26e1820409a9dfccd8ea6a30f49 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:44.374576 2503 consensus_queue.cc:1035] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:44.374737 4010 consensus_queue.cc:1035] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:44.376550 4662 raft_consensus.cc:2932] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:44.376895 30406 raft_consensus.cc:2932] T 588dd26e1820409a9dfccd8ea6a30f49 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:44.378777 30277 raft_consensus.cc:2932] T 588dd26e1820409a9dfccd8ea6a30f49 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:44.379540 29866 catalog_manager.cc:5484] T 588dd26e1820409a9dfccd8ea6a30f49 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:44.382583 32664 tablet_service.cc:1514] Processing DeleteTablet for tablet 588dd26e1820409a9dfccd8ea6a30f49 with delete_type TABLET_DATA_TOMBSTONED (TS db935501e9104de5b15c8780681d6c38 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:44.382921 4665 tablet_replica.cc:330] T 588dd26e1820409a9dfccd8ea6a30f49 P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:44.383050 4665 raft_consensus.cc:2229] T 588dd26e1820409a9dfccd8ea6a30f49 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.383167 4665 raft_consensus.cc:2258] T 588dd26e1820409a9dfccd8ea6a30f49 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.383565 4665 ts_tablet_manager.cc:1875] T 588dd26e1820409a9dfccd8ea6a30f49 P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:44.385304 29865 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:44.385413 29865 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:44.385625 4665 ts_tablet_manager.cc:1888] T 588dd26e1820409a9dfccd8ea6a30f49 P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:09:44.385684 4665 log.cc:1196] T 588dd26e1820409a9dfccd8ea6a30f49 P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/588dd26e1820409a9dfccd8ea6a30f49
I20240501 14:09:44.386226 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet 588dd26e1820409a9dfccd8ea6a30f49 (table rebalancer_extra_table_8 [id=8ff8f244c547410f853c771c11ecdd1e]) successfully deleted
I20240501 14:09:44.388027 29865 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 558c5e9e95ff4086acc1bde63a4c7e8d
I20240501 14:09:44.388595 327 tablet_service.cc:1514] Processing DeleteTablet for tablet 558c5e9e95ff4086acc1bde63a4c7e8d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:44 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:44.388622 32400 tablet_service.cc:1514] Processing DeleteTablet for tablet 558c5e9e95ff4086acc1bde63a4c7e8d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:44 UTC) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:44.388813 32532 tablet_service.cc:1514] Processing DeleteTablet for tablet 558c5e9e95ff4086acc1bde63a4c7e8d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:44 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:44.388885 4670 tablet_replica.cc:330] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:44.389042 4670 raft_consensus.cc:2229] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.389101 4671 tablet_replica.cc:330] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:44.389258 4671 raft_consensus.cc:2229] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:44.389266 4670 raft_consensus.cc:2258] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.389488 4671 raft_consensus.cc:2258] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.389714 4669 tablet_replica.cc:330] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:44.389859 4670 ts_tablet_manager.cc:1875] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:44.389959 4669 raft_consensus.cc:2229] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.390053 4671 ts_tablet_manager.cc:1875] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:44.390087 4669 raft_consensus.cc:2258] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.390578 4669 ts_tablet_manager.cc:1875] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:44.392735 4671 ts_tablet_manager.cc:1888] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:44.392870 4671 log.cc:1196] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/558c5e9e95ff4086acc1bde63a4c7e8d
I20240501 14:09:44.393368 4671 ts_tablet_manager.cc:1909] T 558c5e9e95ff4086acc1bde63a4c7e8d P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:44.393407 4669 ts_tablet_manager.cc:1888] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:44.393440 4670 ts_tablet_manager.cc:1888] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:44.393473 4669 log.cc:1196] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/558c5e9e95ff4086acc1bde63a4c7e8d
I20240501 14:09:44.393499 4670 log.cc:1196] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/558c5e9e95ff4086acc1bde63a4c7e8d
I20240501 14:09:44.393715 4669 ts_tablet_manager.cc:1909] T 558c5e9e95ff4086acc1bde63a4c7e8d P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:09:44.393759 4670 ts_tablet_manager.cc:1909] T 558c5e9e95ff4086acc1bde63a4c7e8d P a91029b90bf9415a89a2f70736759ad3: Deleting consensus metadata
I20240501 14:09:44.394076 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet 558c5e9e95ff4086acc1bde63a4c7e8d (table rebalancer_dynamic_table_new_name [id=02ebac92b4574702a256ff45701b948d]) successfully deleted
I20240501 14:09:44.394327 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 558c5e9e95ff4086acc1bde63a4c7e8d (table rebalancer_dynamic_table_new_name [id=02ebac92b4574702a256ff45701b948d]) successfully deleted
I20240501 14:09:44.394460 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 558c5e9e95ff4086acc1bde63a4c7e8d (table rebalancer_dynamic_table_new_name [id=02ebac92b4574702a256ff45701b948d]) successfully deleted
I20240501 14:09:44.413637 3042 raft_consensus.cc:482] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:44.413743 3042 raft_consensus.cc:504] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:44.414137 3042 leader_election.cc:288] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:44.414454 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81750b74ac3f48e082e6e6d62ef8f066" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:44.414458 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81750b74ac3f48e082e6e6d62ef8f066" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:44.414567 30277 raft_consensus.cc:2454] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:44.414567 32416 raft_consensus.cc:2454] T 81750b74ac3f48e082e6e6d62ef8f066 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:44.414837 32620 leader_election.cc:302] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:44.415138 3042 raft_consensus.cc:2781] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:44.415185 3042 raft_consensus.cc:482] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:44.415236 3042 raft_consensus.cc:3037] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.416061 3042 raft_consensus.cc:504] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:44.416426 3042 leader_election.cc:288] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:44.416775 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81750b74ac3f48e082e6e6d62ef8f066" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:44.416882 30277 raft_consensus.cc:3037] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.417789 30277 raft_consensus.cc:2454] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:44.418222 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81750b74ac3f48e082e6e6d62ef8f066" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:44.418340 32416 raft_consensus.cc:3037] T 81750b74ac3f48e082e6e6d62ef8f066 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.418671 32620 leader_election.cc:302] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:44.418864 3042 raft_consensus.cc:2781] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:44.418999 3042 raft_consensus.cc:686] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Becoming Leader. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
I20240501 14:09:44.419142 3042 consensus_queue.cc:237] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:44.419363 32416 raft_consensus.cc:2454] T 81750b74ac3f48e082e6e6d62ef8f066 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:44.422255 29865 catalog_manager.cc:5484] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db935501e9104de5b15c8780681d6c38 (127.22.252.135). New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:44.490469 29865 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:44.496677 327 tablet_service.cc:1467] Processing CreateTablet for tablet f1f4cde46ed848559c762f56a56c8e38 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=361b96c16b044b86be69ee22e85bf426]), partition=
I20240501 14:09:44.496991 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f4cde46ed848559c762f56a56c8e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.499259 4642 tablet_bootstrap.cc:492] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:44.499485 32532 tablet_service.cc:1467] Processing CreateTablet for tablet f1f4cde46ed848559c762f56a56c8e38 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=361b96c16b044b86be69ee22e85bf426]), partition=
I20240501 14:09:44.499752 32532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f4cde46ed848559c762f56a56c8e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.500160 4642 tablet_bootstrap.cc:654] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.501315 4642 tablet_bootstrap.cc:492] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:44.501377 4642 ts_tablet_manager.cc:1367] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:44.501822 4641 tablet_bootstrap.cc:492] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:44.501807 4642 raft_consensus.cc:348] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:44.501914 4642 raft_consensus.cc:374] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.501942 4642 raft_consensus.cc:729] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.502071 4642 consensus_queue.cc:260] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:44.502372 4642 ts_tablet_manager.cc:1398] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:44.502753 4641 tablet_bootstrap.cc:654] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.503566 32664 tablet_service.cc:1467] Processing CreateTablet for tablet f1f4cde46ed848559c762f56a56c8e38 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=361b96c16b044b86be69ee22e85bf426]), partition=
I20240501 14:09:44.503808 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f4cde46ed848559c762f56a56c8e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.503871 4641 tablet_bootstrap.cc:492] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:44.503950 4641 ts_tablet_manager.cc:1367] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:44.504415 4641 raft_consensus.cc:348] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:44.504508 4641 raft_consensus.cc:374] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.504535 4641 raft_consensus.cc:729] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.504642 4641 consensus_queue.cc:260] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:44.504954 4641 ts_tablet_manager.cc:1398] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:44.505981 4640 tablet_bootstrap.cc:492] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:44.506773 4640 tablet_bootstrap.cc:654] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.509088 4640 tablet_bootstrap.cc:492] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:44.509158 4640 ts_tablet_manager.cc:1367] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:44.509564 4640 raft_consensus.cc:348] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:44.509655 4640 raft_consensus.cc:374] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.509687 4640 raft_consensus.cc:729] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.509814 4640 consensus_queue.cc:260] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:44.509877 4371 raft_consensus.cc:482] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:44.509979 4371 raft_consensus.cc:504] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:44.510103 4640 ts_tablet_manager.cc:1398] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:44.510382 4371 leader_election.cc:288] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:44.510774 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50dc380460bb4f859d25ec77291faffb" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:44.510888 32553 raft_consensus.cc:2454] T 50dc380460bb4f859d25ec77291faffb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:44.510950 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50dc380460bb4f859d25ec77291faffb" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:44.511081 349 raft_consensus.cc:2454] T 50dc380460bb4f859d25ec77291faffb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:44.511195 30344 leader_election.cc:302] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:44.511359 4371 raft_consensus.cc:2781] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:44.511425 4371 raft_consensus.cc:482] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:44.511477 4371 raft_consensus.cc:3037] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.512445 4371 raft_consensus.cc:504] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:44.512782 4371 leader_election.cc:288] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:44.513072 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50dc380460bb4f859d25ec77291faffb" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:44.513098 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50dc380460bb4f859d25ec77291faffb" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:44.513168 349 raft_consensus.cc:3037] T 50dc380460bb4f859d25ec77291faffb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.513195 32553 raft_consensus.cc:3037] T 50dc380460bb4f859d25ec77291faffb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.514113 349 raft_consensus.cc:2454] T 50dc380460bb4f859d25ec77291faffb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:44.514149 32553 raft_consensus.cc:2454] T 50dc380460bb4f859d25ec77291faffb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:44.514410 30343 leader_election.cc:302] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:44.514716 4371 raft_consensus.cc:2781] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:44.514806 4371 raft_consensus.cc:686] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:44.514945 4371 consensus_queue.cc:237] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:44.516696 29865 catalog_manager.cc:5484] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:44.611172 4625 raft_consensus.cc:482] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:44.611297 4625 raft_consensus.cc:504] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:44.611688 4625 leader_election.cc:288] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:44.612015 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f4cde46ed848559c762f56a56c8e38" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:44.612164 32553 raft_consensus.cc:2454] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:44.612149 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f4cde46ed848559c762f56a56c8e38" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:44.612313 32684 raft_consensus.cc:2454] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:44.612490 32753 leader_election.cc:302] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:44.612710 4625 raft_consensus.cc:2781] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:44.612776 4625 raft_consensus.cc:482] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:44.612836 4625 raft_consensus.cc:3037] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.613838 4625 raft_consensus.cc:504] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:44.614130 4625 leader_election.cc:288] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:44.614393 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f4cde46ed848559c762f56a56c8e38" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:44.614393 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f4cde46ed848559c762f56a56c8e38" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:44.614509 32553 raft_consensus.cc:3037] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.614513 32684 raft_consensus.cc:3037] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.615514 32684 raft_consensus.cc:2454] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:44.615514 32553 raft_consensus.cc:2454] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:44.615890 32750 leader_election.cc:302] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:44.616031 4625 raft_consensus.cc:2781] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:44.616114 4625 raft_consensus.cc:686] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:44.616312 4625 consensus_queue.cc:237] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:44.618628 29866 catalog_manager.cc:5484] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:44.702178 2864 raft_consensus.cc:482] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:44.702302 2864 raft_consensus.cc:504] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:44.702733 2864 leader_election.cc:288] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:44.703014 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a277d983e2a47fda9410e1fb8c31221" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:44.703033 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a277d983e2a47fda9410e1fb8c31221" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:44.703150 32684 raft_consensus.cc:2454] T 1a277d983e2a47fda9410e1fb8c31221 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:44.703182 30146 raft_consensus.cc:2454] T 1a277d983e2a47fda9410e1fb8c31221 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:44.703474 30341 leader_election.cc:302] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:44.703734 2864 raft_consensus.cc:2781] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:44.703786 2864 raft_consensus.cc:482] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:44.703826 2864 raft_consensus.cc:3037] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.704694 2864 raft_consensus.cc:504] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:44.705003 2864 leader_election.cc:288] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:44.705258 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a277d983e2a47fda9410e1fb8c31221" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:44.705269 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a277d983e2a47fda9410e1fb8c31221" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:44.705350 32684 raft_consensus.cc:3037] T 1a277d983e2a47fda9410e1fb8c31221 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.705361 30146 raft_consensus.cc:3037] T 1a277d983e2a47fda9410e1fb8c31221 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.706137 32684 raft_consensus.cc:2454] T 1a277d983e2a47fda9410e1fb8c31221 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:44.706248 30146 raft_consensus.cc:2454] T 1a277d983e2a47fda9410e1fb8c31221 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:44.706395 30341 leader_election.cc:302] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:44.706559 2864 raft_consensus.cc:2781] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:44.706670 2864 raft_consensus.cc:686] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:44.707001 2864 consensus_queue.cc:237] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:44.708951 29866 catalog_manager.cc:5484] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:44.727494 29866 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:44.733812 32532 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:44.737596 32684 raft_consensus.cc:1261] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:44.737993 32553 raft_consensus.cc:1261] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:44.738268 4625 consensus_queue.cc:1035] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:44.738427 4167 consensus_queue.cc:1035] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:44.738956 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:44.741385 29866 catalog_manager.cc:5576] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497) does not have the latest schema for tablet f1f4cde46ed848559c762f56a56c8e38 (table rebalancer_dynamic_table [id=361b96c16b044b86be69ee22e85bf426]). Expected version 1 got 0
I20240501 14:09:44.741459 4277 tablet.cc:1722] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:44.741475 4534 tablet.cc:1722] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:44.741765 4534 tablet.cc:1620] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:44.741768 4277 tablet.cc:1620] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:44.746897 29866 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet f1f4cde46ed848559c762f56a56c8e38 (table rebalancer_dynamic_table [id=361b96c16b044b86be69ee22e85bf426]). Expected version 1 got 0
I20240501 14:09:44.748896 29995 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:44.751924 30388 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:44.748286 29871 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=361b96c16b044b86be69ee22e85bf426] alter complete (version 1)
I20240501 14:09:44.756004 32664 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:44.759418 327 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:44.762209 30124 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:44.796382 32400 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:44.806977 29866 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet f1f4cde46ed848559c762f56a56c8e38 (table rebalancer_dynamic_table [id=361b96c16b044b86be69ee22e85bf426]). Expected version 1 got 0
I20240501 14:09:44.809465 4757 tablet.cc:1722] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:44.809831 4757 tablet.cc:1620] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:44.846331 30277 consensus_queue.cc:237] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:44.847436 32553 raft_consensus.cc:1261] T c5d406c64b2e432a88e8c675b91dd144 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.847496 30406 raft_consensus.cc:1261] T c5d406c64b2e432a88e8c675b91dd144 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:44.847906 4774 consensus_queue.cc:1035] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.848181 4242 consensus_queue.cc:1035] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:44.849831 4242 raft_consensus.cc:2932] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:44.850284 32553 raft_consensus.cc:2932] T c5d406c64b2e432a88e8c675b91dd144 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:44.850318 30406 raft_consensus.cc:2932] T c5d406c64b2e432a88e8c675b91dd144 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:44.853097 29871 catalog_manager.cc:5484] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: config changed from index 4 to 5, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:44.856670 29995 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d406c64b2e432a88e8c675b91dd144 with delete_type TABLET_DATA_TOMBSTONED (TS 93f2518fbadf4f9991cda93afacb363e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:44.857134 4783 tablet_replica.cc:330] T c5d406c64b2e432a88e8c675b91dd144 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:44.857180 29871 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:44.857358 4783 raft_consensus.cc:2229] T c5d406c64b2e432a88e8c675b91dd144 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.857525 4783 raft_consensus.cc:2258] T c5d406c64b2e432a88e8c675b91dd144 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.858055 4783 ts_tablet_manager.cc:1875] T c5d406c64b2e432a88e8c675b91dd144 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:44.862452 4277 tablet.cc:1722] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:44.862768 4277 tablet.cc:1620] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:44.863412 4757 tablet.cc:1722] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:44.863551 4534 tablet.cc:1722] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:44.863711 4757 tablet.cc:1620] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:44.863816 4534 tablet.cc:1620] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:44.863937 4783 ts_tablet_manager.cc:1888] T c5d406c64b2e432a88e8c675b91dd144 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:44.864017 4783 log.cc:1196] T c5d406c64b2e432a88e8c675b91dd144 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/c5d406c64b2e432a88e8c675b91dd144
I20240501 14:09:44.864547 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet c5d406c64b2e432a88e8c675b91dd144 (table rebalance_test_table_7 [id=8f3109027b0c482d89eb86da7f7ed641]) successfully deleted
I20240501 14:09:44.868573 29866 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=361b96c16b044b86be69ee22e85bf426] alter complete (version 2)
I20240501 14:09:44.891681 30146 consensus_queue.cc:237] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:44.893009 32684 raft_consensus.cc:1261] T c7df6b8d76714eebbeacc85f403d3405 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:44.893116 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c7df6b8d76714eebbeacc85f403d3405. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.893361 4805 consensus_queue.cc:1035] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:44.893460 32416 raft_consensus.cc:1261] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:44.893764 4805 consensus_queue.cc:1035] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:44.895532 2969 raft_consensus.cc:2932] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:44.895941 32416 raft_consensus.cc:2932] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:44.898576 29865 catalog_manager.cc:5484] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 reported cstate change: config changed from index -1 to 2, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:44.899391 32684 raft_consensus.cc:2932] T c7df6b8d76714eebbeacc85f403d3405 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:44.915877 32416 consensus_queue.cc:237] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:44.917129 32553 raft_consensus.cc:1261] T 7a6a29261186429cbc05b1819999f986 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:44.917178 349 raft_consensus.cc:1261] T 7a6a29261186429cbc05b1819999f986 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:44.917591 4826 consensus_queue.cc:1035] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:44.917755 4825 consensus_queue.cc:1035] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:09:44.917953 32356 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): Couldn't send request to peer d1c217469ae4478ebd49e4ed9f7352d7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a6a29261186429cbc05b1819999f986. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.919881 4825 raft_consensus.cc:2932] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:44.920342 349 raft_consensus.cc:2932] T 7a6a29261186429cbc05b1819999f986 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:44.922127 32553 raft_consensus.cc:2932] T 7a6a29261186429cbc05b1819999f986 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:44.923486 29866 catalog_manager.cc:5484] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:44.925595 3042 consensus_queue.cc:1035] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:44.929662 4829 consensus_queue.cc:1035] T 81750b74ac3f48e082e6e6d62ef8f066 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:44.945456 30013 consensus_queue.cc:237] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
W20240501 14:09:44.946867 29950 consensus_peers.cc:485] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e -> Peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): Couldn't send request to peer a91029b90bf9415a89a2f70736759ad3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15c98173a71d4f73b8f6af24ad01d7c4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.946856 30277 raft_consensus.cc:1261] T 15c98173a71d4f73b8f6af24ad01d7c4 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:44.946946 30146 raft_consensus.cc:1261] T 15c98173a71d4f73b8f6af24ad01d7c4 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:44.947175 4852 consensus_queue.cc:1035] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:44.947347 3455 consensus_queue.cc:1035] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:44.948943 4852 raft_consensus.cc:2932] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:44.949533 30146 raft_consensus.cc:2932] T 15c98173a71d4f73b8f6af24ad01d7c4 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:44.951620 30277 raft_consensus.cc:2932] T 15c98173a71d4f73b8f6af24ad01d7c4 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:44.951857 29871 catalog_manager.cc:5484] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: config changed from index -1 to 4, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:44.971298 29871 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:44.971546 32415 consensus_queue.cc:237] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
W20240501 14:09:44.973320 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Couldn't send request to peer d35c16dcb506426993811bb2519d7db3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c5c502325fb475f8f050ccad34feb2a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:44.973757 32684 raft_consensus.cc:1261] T 2c5c502325fb475f8f050ccad34feb2a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:44.974238 32553 raft_consensus.cc:1261] T 2c5c502325fb475f8f050ccad34feb2a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:44.974442 4825 consensus_queue.cc:1035] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:44.974594 4662 consensus_queue.cc:1035] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:44.976329 4662 raft_consensus.cc:2932] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:44.976735 32684 raft_consensus.cc:2932] T 2c5c502325fb475f8f050ccad34feb2a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:44.978871 32553 raft_consensus.cc:2932] T 2c5c502325fb475f8f050ccad34feb2a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:44.981289 29865 catalog_manager.cc:5484] T 2c5c502325fb475f8f050ccad34feb2a P db935501e9104de5b15c8780681d6c38 reported cstate change: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:44.987542 4277 tablet.cc:1722] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:44.987953 4534 tablet.cc:1722] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:44.989023 4757 tablet.cc:1722] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:44.993405 29866 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=361b96c16b044b86be69ee22e85bf426] alter complete (version 3)
I20240501 14:09:45.000919 4371 consensus_queue.cc:1035] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.004164 4371 consensus_queue.cc:1035] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.025151 30406 consensus_queue.cc:237] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:45.026516 32553 raft_consensus.cc:1261] T e122adbba7c749cd93c53fa07a8a8b84 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:45.026604 30013 raft_consensus.cc:1261] T e122adbba7c749cd93c53fa07a8a8b84 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:45.026820 30343 consensus_peers.cc:485] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): Couldn't send request to peer 9ecd907cf5634ac98f3f54f3dbec6849. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e122adbba7c749cd93c53fa07a8a8b84. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:45.027010 2864 consensus_queue.cc:1035] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:45.027535 2864 consensus_queue.cc:1035] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:45.029201 4371 raft_consensus.cc:2932] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:45.029616 32553 raft_consensus.cc:2932] T e122adbba7c749cd93c53fa07a8a8b84 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:45.030997 30013 raft_consensus.cc:2932] T e122adbba7c749cd93c53fa07a8a8b84 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:45.032106 29868 catalog_manager.cc:5484] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index -1 to 2, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:45.086596 4740 consensus_queue.cc:1035] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.095585 4371 consensus_queue.cc:1035] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.097504 29872 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:45.097620 29872 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:45.100309 29872 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet f1f4cde46ed848559c762f56a56c8e38
I20240501 14:09:45.100883 32532 tablet_service.cc:1514] Processing DeleteTablet for tablet f1f4cde46ed848559c762f56a56c8e38 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:45 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:45.100937 32664 tablet_service.cc:1514] Processing DeleteTablet for tablet f1f4cde46ed848559c762f56a56c8e38 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:45 UTC) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:45.101366 327 tablet_service.cc:1514] Processing DeleteTablet for tablet f1f4cde46ed848559c762f56a56c8e38 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:45 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:45.101682 4942 tablet_replica.cc:330] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:45.101792 4941 tablet_replica.cc:330] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:45.101902 4942 raft_consensus.cc:2229] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:45.101945 4941 raft_consensus.cc:2229] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.102177 4940 tablet_replica.cc:330] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:45.102190 4942 raft_consensus.cc:2258] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.102380 4940 raft_consensus.cc:2229] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.102560 4941 raft_consensus.cc:2258] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.102725 4940 raft_consensus.cc:2258] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.102828 4942 ts_tablet_manager.cc:1875] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:45.108803 4941 ts_tablet_manager.cc:1875] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:45.109042 4940 ts_tablet_manager.cc:1875] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:45.110064 4942 ts_tablet_manager.cc:1888] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:45.110146 4942 log.cc:1196] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/f1f4cde46ed848559c762f56a56c8e38
I20240501 14:09:45.110474 4942 ts_tablet_manager.cc:1909] T f1f4cde46ed848559c762f56a56c8e38 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:09:45.111452 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet f1f4cde46ed848559c762f56a56c8e38 (table rebalancer_dynamic_table_new_name [id=361b96c16b044b86be69ee22e85bf426]) successfully deleted
I20240501 14:09:45.116104 4940 ts_tablet_manager.cc:1888] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:45.116189 4940 log.cc:1196] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/f1f4cde46ed848559c762f56a56c8e38
I20240501 14:09:45.116444 4941 ts_tablet_manager.cc:1888] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:45.116524 4941 log.cc:1196] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/f1f4cde46ed848559c762f56a56c8e38
I20240501 14:09:45.116838 4940 ts_tablet_manager.cc:1909] T f1f4cde46ed848559c762f56a56c8e38 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:45.117172 4941 ts_tablet_manager.cc:1909] T f1f4cde46ed848559c762f56a56c8e38 P db935501e9104de5b15c8780681d6c38: Deleting consensus metadata
I20240501 14:09:45.118018 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet f1f4cde46ed848559c762f56a56c8e38 (table rebalancer_dynamic_table_new_name [id=361b96c16b044b86be69ee22e85bf426]) successfully deleted
I20240501 14:09:45.118753 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet f1f4cde46ed848559c762f56a56c8e38 (table rebalancer_dynamic_table_new_name [id=361b96c16b044b86be69ee22e85bf426]) successfully deleted
I20240501 14:09:45.130411 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:45.133443 32532 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:45.134611 29995 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:45.139183 32400 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:45.141644 32664 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:45.153375 30388 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:45.154975 30124 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:45.192070 327 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:45.202579 29873 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:45.213510 32400 tablet_service.cc:1467] Processing CreateTablet for tablet e6af3f1cff644f4091e530d81e958ffa (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0dba7d13b364430ab8806c35258df29d]), partition=
I20240501 14:09:45.213640 32532 tablet_service.cc:1467] Processing CreateTablet for tablet e6af3f1cff644f4091e530d81e958ffa (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0dba7d13b364430ab8806c35258df29d]), partition=
I20240501 14:09:45.213794 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6af3f1cff644f4091e530d81e958ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.213861 32532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6af3f1cff644f4091e530d81e958ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.216055 32664 tablet_service.cc:1467] Processing CreateTablet for tablet e6af3f1cff644f4091e530d81e958ffa (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0dba7d13b364430ab8806c35258df29d]), partition=
I20240501 14:09:45.216648 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6af3f1cff644f4091e530d81e958ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.219128 4956 tablet_bootstrap.cc:492] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:45.220343 4956 tablet_bootstrap.cc:654] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.220577 4954 tablet_bootstrap.cc:492] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:45.221028 4955 tablet_bootstrap.cc:492] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:45.221664 4954 tablet_bootstrap.cc:654] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.221969 4955 tablet_bootstrap.cc:654] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.223062 4955 tablet_bootstrap.cc:492] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:45.223150 4955 ts_tablet_manager.cc:1367] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:45.223479 4956 tablet_bootstrap.cc:492] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:45.223565 4956 ts_tablet_manager.cc:1367] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:45.223677 4955 raft_consensus.cc:348] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:45.223791 4955 raft_consensus.cc:374] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.223853 4955 raft_consensus.cc:729] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.223984 4955 consensus_queue.cc:260] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:45.224054 4954 tablet_bootstrap.cc:492] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:45.224217 4954 ts_tablet_manager.cc:1367] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:45.224606 4956 raft_consensus.cc:348] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:45.225049 4955 ts_tablet_manager.cc:1398] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:45.225239 4956 raft_consensus.cc:374] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.225220 4954 raft_consensus.cc:348] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:45.225335 4956 raft_consensus.cc:729] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.225383 4954 raft_consensus.cc:374] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.225531 4956 consensus_queue.cc:260] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:45.226174 4956 ts_tablet_manager.cc:1398] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:45.225478 4954 raft_consensus.cc:729] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.228215 4954 consensus_queue.cc:260] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:45.232164 4954 ts_tablet_manager.cc:1398] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240501 14:09:45.234855 29869 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:45.241792 327 tablet_service.cc:1467] Processing CreateTablet for tablet 609a7f5fccdf4a8a9f854d9007412d33 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=2de4e5e882624568a22539fc9f48ad52]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.241976 32532 tablet_service.cc:1467] Processing CreateTablet for tablet 27564c531f6147aeac4809082ffe2402 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=2de4e5e882624568a22539fc9f48ad52]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.242092 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609a7f5fccdf4a8a9f854d9007412d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.242215 32532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27564c531f6147aeac4809082ffe2402. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.242341 32534 tablet_service.cc:1467] Processing CreateTablet for tablet aa8ba266a8d0406bb93d6c578fe6c7cb (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=2de4e5e882624568a22539fc9f48ad52]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.242559 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa8ba266a8d0406bb93d6c578fe6c7cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.244575 4954 tablet_bootstrap.cc:492] T 27564c531f6147aeac4809082ffe2402 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:45.244529 29995 tablet_service.cc:1467] Processing CreateTablet for tablet 609a7f5fccdf4a8a9f854d9007412d33 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=2de4e5e882624568a22539fc9f48ad52]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.244789 29995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609a7f5fccdf4a8a9f854d9007412d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.245170 30256 tablet_service.cc:1467] Processing CreateTablet for tablet aa8ba266a8d0406bb93d6c578fe6c7cb (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=2de4e5e882624568a22539fc9f48ad52]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.245432 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa8ba266a8d0406bb93d6c578fe6c7cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.245509 4954 tablet_bootstrap.cc:654] T 27564c531f6147aeac4809082ffe2402 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.246966 4972 tablet_bootstrap.cc:492] T 609a7f5fccdf4a8a9f854d9007412d33 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:45.247612 4973 tablet_bootstrap.cc:492] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:45.247867 4972 tablet_bootstrap.cc:654] T 609a7f5fccdf4a8a9f854d9007412d33 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.248893 4973 tablet_bootstrap.cc:654] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.249596 30124 tablet_service.cc:1467] Processing CreateTablet for tablet 609a7f5fccdf4a8a9f854d9007412d33 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=2de4e5e882624568a22539fc9f48ad52]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.249886 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609a7f5fccdf4a8a9f854d9007412d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.249924 4973 tablet_bootstrap.cc:492] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:45.250006 4973 ts_tablet_manager.cc:1367] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:45.250396 4973 raft_consensus.cc:348] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:45.250489 4973 raft_consensus.cc:374] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.250550 4973 raft_consensus.cc:729] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.250500 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 27564c531f6147aeac4809082ffe2402 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=2de4e5e882624568a22539fc9f48ad52]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.250661 4973 consensus_queue.cc:260] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:45.250768 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27564c531f6147aeac4809082ffe2402. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.250905 4973 ts_tablet_manager.cc:1398] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.251313 32664 tablet_service.cc:1467] Processing CreateTablet for tablet aa8ba266a8d0406bb93d6c578fe6c7cb (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=2de4e5e882624568a22539fc9f48ad52]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.251569 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa8ba266a8d0406bb93d6c578fe6c7cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.253000 4977 tablet_bootstrap.cc:492] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:45.253643 4956 tablet_bootstrap.cc:492] T aa8ba266a8d0406bb93d6c578fe6c7cb P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:45.254063 4977 tablet_bootstrap.cc:654] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.254621 4956 tablet_bootstrap.cc:654] T aa8ba266a8d0406bb93d6c578fe6c7cb P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.255893 4956 tablet_bootstrap.cc:492] T aa8ba266a8d0406bb93d6c578fe6c7cb P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:45.255990 4956 ts_tablet_manager.cc:1367] T aa8ba266a8d0406bb93d6c578fe6c7cb P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:45.256501 4956 raft_consensus.cc:348] T aa8ba266a8d0406bb93d6c578fe6c7cb P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:45.256599 4956 raft_consensus.cc:374] T aa8ba266a8d0406bb93d6c578fe6c7cb P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.256629 4956 raft_consensus.cc:729] T aa8ba266a8d0406bb93d6c578fe6c7cb P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.256736 4956 consensus_queue.cc:260] T aa8ba266a8d0406bb93d6c578fe6c7cb P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:45.257048 4956 ts_tablet_manager.cc:1398] T aa8ba266a8d0406bb93d6c578fe6c7cb P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:45.257778 29995 tablet_service.cc:1467] Processing CreateTablet for tablet 27564c531f6147aeac4809082ffe2402 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=2de4e5e882624568a22539fc9f48ad52]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.258055 29995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27564c531f6147aeac4809082ffe2402. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.259690 4976 tablet_bootstrap.cc:492] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:45.261190 4980 tablet_bootstrap.cc:492] T 609a7f5fccdf4a8a9f854d9007412d33 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:45.262293 4980 tablet_bootstrap.cc:654] T 609a7f5fccdf4a8a9f854d9007412d33 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.263425 4980 tablet_bootstrap.cc:492] T 609a7f5fccdf4a8a9f854d9007412d33 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:45.263516 4980 ts_tablet_manager.cc:1367] T 609a7f5fccdf4a8a9f854d9007412d33 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:45.264499 4980 raft_consensus.cc:348] T 609a7f5fccdf4a8a9f854d9007412d33 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:45.264674 4980 raft_consensus.cc:374] T 609a7f5fccdf4a8a9f854d9007412d33 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.265064 4980 raft_consensus.cc:729] T 609a7f5fccdf4a8a9f854d9007412d33 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.265153 4976 tablet_bootstrap.cc:654] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.265255 4954 tablet_bootstrap.cc:492] T 27564c531f6147aeac4809082ffe2402 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:45.265349 4954 ts_tablet_manager.cc:1367] T 27564c531f6147aeac4809082ffe2402 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.001s
I20240501 14:09:45.265812 4954 raft_consensus.cc:348] T 27564c531f6147aeac4809082ffe2402 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:45.265933 4954 raft_consensus.cc:374] T 27564c531f6147aeac4809082ffe2402 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.265966 4954 raft_consensus.cc:729] T 27564c531f6147aeac4809082ffe2402 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.266081 4954 consensus_queue.cc:260] T 27564c531f6147aeac4809082ffe2402 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:45.266399 4954 ts_tablet_manager.cc:1398] T 27564c531f6147aeac4809082ffe2402 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 14:09:45.266559 4954 tablet_bootstrap.cc:492] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:45.265272 4980 consensus_queue.cc:260] T 609a7f5fccdf4a8a9f854d9007412d33 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:45.267464 4954 tablet_bootstrap.cc:654] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.268460 4980 ts_tablet_manager.cc:1398] T 609a7f5fccdf4a8a9f854d9007412d33 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:45.269381 4977 tablet_bootstrap.cc:492] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:45.269429 4976 tablet_bootstrap.cc:492] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:45.269482 4977 ts_tablet_manager.cc:1367] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 14:09:45.269507 4976 ts_tablet_manager.cc:1367] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 14:09:45.269925 4977 raft_consensus.cc:348] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:45.270038 4977 raft_consensus.cc:374] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.270107 4977 raft_consensus.cc:729] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.270094 4976 raft_consensus.cc:348] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:45.270198 4976 raft_consensus.cc:374] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.270638 4954 tablet_bootstrap.cc:492] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:45.270713 4954 ts_tablet_manager.cc:1367] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:09:45.271170 4954 raft_consensus.cc:348] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:45.271291 4954 raft_consensus.cc:374] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.271322 4954 raft_consensus.cc:729] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.271445 4954 consensus_queue.cc:260] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:45.270238 4976 raft_consensus.cc:729] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.271728 4976 consensus_queue.cc:260] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:45.272137 4976 ts_tablet_manager.cc:1398] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:45.272883 4972 tablet_bootstrap.cc:492] T 609a7f5fccdf4a8a9f854d9007412d33 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:45.272981 4972 ts_tablet_manager.cc:1367] T 609a7f5fccdf4a8a9f854d9007412d33 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240501 14:09:45.273228 4954 ts_tablet_manager.cc:1398] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:09:45.273404 4977 consensus_queue.cc:260] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:45.273450 4972 raft_consensus.cc:348] T 609a7f5fccdf4a8a9f854d9007412d33 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:45.273556 4972 raft_consensus.cc:374] T 609a7f5fccdf4a8a9f854d9007412d33 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.273591 4972 raft_consensus.cc:729] T 609a7f5fccdf4a8a9f854d9007412d33 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.273697 4972 consensus_queue.cc:260] T 609a7f5fccdf4a8a9f854d9007412d33 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:45.274044 4972 ts_tablet_manager.cc:1398] T 609a7f5fccdf4a8a9f854d9007412d33 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:45.274202 4972 tablet_bootstrap.cc:492] T 27564c531f6147aeac4809082ffe2402 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:45.274423 4094 raft_consensus.cc:482] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:45.274499 4094 raft_consensus.cc:504] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:45.274670 4977 ts_tablet_manager.cc:1398] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:45.274888 4094 leader_election.cc:288] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:45.275127 4972 tablet_bootstrap.cc:654] T 27564c531f6147aeac4809082ffe2402 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.275213 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609a7f5fccdf4a8a9f854d9007412d33" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:45.275317 30013 raft_consensus.cc:2454] T 609a7f5fccdf4a8a9f854d9007412d33 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:45.276023 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609a7f5fccdf4a8a9f854d9007412d33" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:45.276098 30081 leader_election.cc:302] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:45.276137 349 raft_consensus.cc:2454] T 609a7f5fccdf4a8a9f854d9007412d33 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:45.276293 4094 raft_consensus.cc:2781] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:45.276338 4094 raft_consensus.cc:482] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:45.276371 4094 raft_consensus.cc:3037] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.277396 4094 raft_consensus.cc:504] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:45.277741 4094 leader_election.cc:288] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:45.278916 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609a7f5fccdf4a8a9f854d9007412d33" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:45.279033 349 raft_consensus.cc:3037] T 609a7f5fccdf4a8a9f854d9007412d33 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.279892 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609a7f5fccdf4a8a9f854d9007412d33" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:45.279970 349 raft_consensus.cc:2454] T 609a7f5fccdf4a8a9f854d9007412d33 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:45.280002 30013 raft_consensus.cc:3037] T 609a7f5fccdf4a8a9f854d9007412d33 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.281054 30013 raft_consensus.cc:2454] T 609a7f5fccdf4a8a9f854d9007412d33 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:45.282135 30081 leader_election.cc:302] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:45.283082 4808 raft_consensus.cc:2781] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:45.283177 4808 raft_consensus.cc:686] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:45.283332 4808 consensus_queue.cc:237] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:45.284865 4988 ts_tablet_manager.cc:902] T c7df6b8d76714eebbeacc85f403d3405 P 5f8a0cfdfdca4148a75686dc88059cf3: Initiating tablet copy from peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:45.285430 4988 tablet_copy_client.cc:308] T c7df6b8d76714eebbeacc85f403d3405 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.130:36457
I20240501 14:09:45.285630 29869 catalog_manager.cc:5484] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:45.286192 4972 tablet_bootstrap.cc:492] T 27564c531f6147aeac4809082ffe2402 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:45.286279 4972 ts_tablet_manager.cc:1367] T 27564c531f6147aeac4809082ffe2402 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20240501 14:09:45.286252 30165 tablet_copy_service.cc:140] P d35c16dcb506426993811bb2519d7db3: Received BeginTabletCopySession request for tablet c7df6b8d76714eebbeacc85f403d3405 from peer 5f8a0cfdfdca4148a75686dc88059cf3 ({username='slave'} at 127.22.252.134:40399)
I20240501 14:09:45.286346 30165 tablet_copy_service.cc:161] P d35c16dcb506426993811bb2519d7db3: Beginning new tablet copy session on tablet c7df6b8d76714eebbeacc85f403d3405 from peer 5f8a0cfdfdca4148a75686dc88059cf3 at {username='slave'} at 127.22.252.134:40399: session id = 5f8a0cfdfdca4148a75686dc88059cf3-c7df6b8d76714eebbeacc85f403d3405
I20240501 14:09:45.286839 4972 raft_consensus.cc:348] T 27564c531f6147aeac4809082ffe2402 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:45.286919 4972 raft_consensus.cc:374] T 27564c531f6147aeac4809082ffe2402 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.286950 4972 raft_consensus.cc:729] T 27564c531f6147aeac4809082ffe2402 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.287076 4972 consensus_queue.cc:260] T 27564c531f6147aeac4809082ffe2402 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:45.287245 30165 tablet_copy_source_session.cc:205] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:45.287328 4972 ts_tablet_manager.cc:1398] T 27564c531f6147aeac4809082ffe2402 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:45.288334 4988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7df6b8d76714eebbeacc85f403d3405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.291088 4988 tablet_copy_client.cc:783] T c7df6b8d76714eebbeacc85f403d3405 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:45.291249 4988 tablet_copy_client.cc:647] T c7df6b8d76714eebbeacc85f403d3405 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:45.292332 4988 tablet_copy_client.cc:523] T c7df6b8d76714eebbeacc85f403d3405 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:45.293670 5001 ts_tablet_manager.cc:902] T 7a6a29261186429cbc05b1819999f986 P d1c217469ae4478ebd49e4ed9f7352d7: Initiating tablet copy from peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:45.293788 4988 tablet_bootstrap.cc:492] T c7df6b8d76714eebbeacc85f403d3405 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:45.294775 5001 tablet_copy_client.cc:308] T 7a6a29261186429cbc05b1819999f986 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.133:46265
I20240501 14:09:45.295437 32443 tablet_copy_service.cc:140] P a91029b90bf9415a89a2f70736759ad3: Received BeginTabletCopySession request for tablet 7a6a29261186429cbc05b1819999f986 from peer d1c217469ae4478ebd49e4ed9f7352d7 ({username='slave'} at 127.22.252.132:53277)
I20240501 14:09:45.295555 32443 tablet_copy_service.cc:161] P a91029b90bf9415a89a2f70736759ad3: Beginning new tablet copy session on tablet 7a6a29261186429cbc05b1819999f986 from peer d1c217469ae4478ebd49e4ed9f7352d7 at {username='slave'} at 127.22.252.132:53277: session id = d1c217469ae4478ebd49e4ed9f7352d7-7a6a29261186429cbc05b1819999f986
I20240501 14:09:45.296510 32443 tablet_copy_source_session.cc:205] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:45.297353 5001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a6a29261186429cbc05b1819999f986. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.297477 4988 tablet_bootstrap.cc:492] T c7df6b8d76714eebbeacc85f403d3405 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:45.297741 4988 tablet_bootstrap.cc:492] T c7df6b8d76714eebbeacc85f403d3405 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap complete.
I20240501 14:09:45.297899 4988 ts_tablet_manager.cc:1367] T c7df6b8d76714eebbeacc85f403d3405 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240501 14:09:45.298383 4988 raft_consensus.cc:348] T c7df6b8d76714eebbeacc85f403d3405 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:45.298566 4988 raft_consensus.cc:729] T c7df6b8d76714eebbeacc85f403d3405 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.298713 4988 consensus_queue.cc:260] T c7df6b8d76714eebbeacc85f403d3405 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:45.299096 4988 ts_tablet_manager.cc:1398] T c7df6b8d76714eebbeacc85f403d3405 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:45.299726 5001 tablet_copy_client.cc:783] T 7a6a29261186429cbc05b1819999f986 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:45.299759 30165 tablet_copy_service.cc:342] P d35c16dcb506426993811bb2519d7db3: Request end of tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-c7df6b8d76714eebbeacc85f403d3405 received from {username='slave'} at 127.22.252.134:40399
I20240501 14:09:45.299825 30165 tablet_copy_service.cc:433] P d35c16dcb506426993811bb2519d7db3: ending tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-c7df6b8d76714eebbeacc85f403d3405 on tablet c7df6b8d76714eebbeacc85f403d3405 with peer 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:09:45.299865 5001 tablet_copy_client.cc:647] T 7a6a29261186429cbc05b1819999f986 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:45.301743 5001 tablet_copy_client.cc:523] T 7a6a29261186429cbc05b1819999f986 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:45.303467 5001 tablet_bootstrap.cc:492] T 7a6a29261186429cbc05b1819999f986 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:45.307915 5001 tablet_bootstrap.cc:492] T 7a6a29261186429cbc05b1819999f986 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:45.308283 5001 tablet_bootstrap.cc:492] T 7a6a29261186429cbc05b1819999f986 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap complete.
I20240501 14:09:45.308498 5001 ts_tablet_manager.cc:1367] T 7a6a29261186429cbc05b1819999f986 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.002s
I20240501 14:09:45.308984 5001 raft_consensus.cc:348] T 7a6a29261186429cbc05b1819999f986 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:45.309190 5001 raft_consensus.cc:729] T 7a6a29261186429cbc05b1819999f986 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.309398 5001 consensus_queue.cc:260] T 7a6a29261186429cbc05b1819999f986 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:45.309796 5001 ts_tablet_manager.cc:1398] T 7a6a29261186429cbc05b1819999f986 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:45.310307 32443 tablet_copy_service.cc:342] P a91029b90bf9415a89a2f70736759ad3: Request end of tablet copy session d1c217469ae4478ebd49e4ed9f7352d7-7a6a29261186429cbc05b1819999f986 received from {username='slave'} at 127.22.252.132:53277
I20240501 14:09:45.310391 32443 tablet_copy_service.cc:433] P a91029b90bf9415a89a2f70736759ad3: ending tablet copy session d1c217469ae4478ebd49e4ed9f7352d7-7a6a29261186429cbc05b1819999f986 on tablet 7a6a29261186429cbc05b1819999f986 with peer d1c217469ae4478ebd49e4ed9f7352d7
I20240501 14:09:45.320963 4468 raft_consensus.cc:482] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:45.321082 4468 raft_consensus.cc:504] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:45.321518 4468 leader_election.cc:288] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:45.321831 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8ba266a8d0406bb93d6c578fe6c7cb" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:45.321966 30277 raft_consensus.cc:2454] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:45.321960 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8ba266a8d0406bb93d6c578fe6c7cb" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:45.322088 32684 raft_consensus.cc:2454] T aa8ba266a8d0406bb93d6c578fe6c7cb P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:45.322276 32489 leader_election.cc:302] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 5f8a0cfdfdca4148a75686dc88059cf3; no voters:
I20240501 14:09:45.322969 4468 raft_consensus.cc:2781] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:45.323033 4468 raft_consensus.cc:482] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:45.323071 4468 raft_consensus.cc:3037] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.324174 4468 raft_consensus.cc:504] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:45.324606 4468 leader_election.cc:288] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:45.324880 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8ba266a8d0406bb93d6c578fe6c7cb" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:45.325001 30277 raft_consensus.cc:3037] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.325371 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8ba266a8d0406bb93d6c578fe6c7cb" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:45.325495 32684 raft_consensus.cc:3037] T aa8ba266a8d0406bb93d6c578fe6c7cb P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.325943 30277 raft_consensus.cc:2454] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:45.326396 32489 leader_election.cc:302] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 5f8a0cfdfdca4148a75686dc88059cf3; no voters:
I20240501 14:09:45.326560 32684 raft_consensus.cc:2454] T aa8ba266a8d0406bb93d6c578fe6c7cb P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:45.326961 4855 raft_consensus.cc:2781] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:45.327049 4855 raft_consensus.cc:686] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:09:45.327165 4855 consensus_queue.cc:237] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:45.329449 29869 catalog_manager.cc:5484] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:45.329962 4371 raft_consensus.cc:482] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:45.330081 4371 raft_consensus.cc:504] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:45.330444 4371 leader_election.cc:288] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:45.330819 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27564c531f6147aeac4809082ffe2402" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:45.330926 32553 raft_consensus.cc:2454] T 27564c531f6147aeac4809082ffe2402 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:45.330919 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27564c531f6147aeac4809082ffe2402" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:45.331053 30013 raft_consensus.cc:2454] T 27564c531f6147aeac4809082ffe2402 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:45.331203 30344 leader_election.cc:302] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:45.331405 4371 raft_consensus.cc:2781] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:45.331458 4371 raft_consensus.cc:482] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:45.331496 4371 raft_consensus.cc:3037] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.332494 4371 raft_consensus.cc:504] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:45.332814 4371 leader_election.cc:288] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:45.333073 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27564c531f6147aeac4809082ffe2402" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:45.333166 32553 raft_consensus.cc:3037] T 27564c531f6147aeac4809082ffe2402 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.333658 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27564c531f6147aeac4809082ffe2402" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:45.333755 30013 raft_consensus.cc:3037] T 27564c531f6147aeac4809082ffe2402 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.334100 32553 raft_consensus.cc:2454] T 27564c531f6147aeac4809082ffe2402 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:45.334378 30344 leader_election.cc:302] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:45.334545 4371 raft_consensus.cc:2781] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:45.334640 4371 raft_consensus.cc:686] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:45.334821 30013 raft_consensus.cc:2454] T 27564c531f6147aeac4809082ffe2402 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:45.334785 4371 consensus_queue.cc:237] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:45.336858 29864 catalog_manager.cc:5484] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:45.453383 5030 ts_tablet_manager.cc:902] T e122adbba7c749cd93c53fa07a8a8b84 P 9ecd907cf5634ac98f3f54f3dbec6849: Initiating tablet copy from peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:45.454038 5030 tablet_copy_client.cc:308] T e122adbba7c749cd93c53fa07a8a8b84 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.132:34357
I20240501 14:09:45.454476 30428 tablet_copy_service.cc:140] P d1c217469ae4478ebd49e4ed9f7352d7: Received BeginTabletCopySession request for tablet e122adbba7c749cd93c53fa07a8a8b84 from peer 9ecd907cf5634ac98f3f54f3dbec6849 ({username='slave'} at 127.22.252.136:38643)
I20240501 14:09:45.454593 30428 tablet_copy_service.cc:161] P d1c217469ae4478ebd49e4ed9f7352d7: Beginning new tablet copy session on tablet e122adbba7c749cd93c53fa07a8a8b84 from peer 9ecd907cf5634ac98f3f54f3dbec6849 at {username='slave'} at 127.22.252.136:38643: session id = 9ecd907cf5634ac98f3f54f3dbec6849-e122adbba7c749cd93c53fa07a8a8b84
I20240501 14:09:45.455439 30428 tablet_copy_source_session.cc:205] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:45.456257 5030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e122adbba7c749cd93c53fa07a8a8b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.458066 5030 tablet_copy_client.cc:783] T e122adbba7c749cd93c53fa07a8a8b84 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:45.458209 5030 tablet_copy_client.cc:647] T e122adbba7c749cd93c53fa07a8a8b84 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:45.459226 5030 tablet_copy_client.cc:523] T e122adbba7c749cd93c53fa07a8a8b84 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:45.460152 5030 tablet_bootstrap.cc:492] T e122adbba7c749cd93c53fa07a8a8b84 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:45.462846 5030 tablet_bootstrap.cc:492] T e122adbba7c749cd93c53fa07a8a8b84 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:45.463054 5030 tablet_bootstrap.cc:492] T e122adbba7c749cd93c53fa07a8a8b84 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap complete.
I20240501 14:09:45.463181 5030 ts_tablet_manager.cc:1367] T e122adbba7c749cd93c53fa07a8a8b84 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:09:45.463501 5030 raft_consensus.cc:348] T e122adbba7c749cd93c53fa07a8a8b84 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:45.463584 5030 raft_consensus.cc:729] T e122adbba7c749cd93c53fa07a8a8b84 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.463699 5030 consensus_queue.cc:260] T e122adbba7c749cd93c53fa07a8a8b84 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:45.463923 5030 ts_tablet_manager.cc:1398] T e122adbba7c749cd93c53fa07a8a8b84 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:45.464326 30428 tablet_copy_service.cc:342] P d1c217469ae4478ebd49e4ed9f7352d7: Request end of tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-e122adbba7c749cd93c53fa07a8a8b84 received from {username='slave'} at 127.22.252.136:38643
I20240501 14:09:45.464390 30428 tablet_copy_service.cc:433] P d1c217469ae4478ebd49e4ed9f7352d7: ending tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-e122adbba7c749cd93c53fa07a8a8b84 on tablet e122adbba7c749cd93c53fa07a8a8b84 with peer 9ecd907cf5634ac98f3f54f3dbec6849
I20240501 14:09:45.504226 4829 raft_consensus.cc:482] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:45.504356 4829 raft_consensus.cc:504] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:45.504676 4829 leader_election.cc:288] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:45.505013 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6af3f1cff644f4091e530d81e958ffa" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:45.505038 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6af3f1cff644f4091e530d81e958ffa" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:45.505156 32415 raft_consensus.cc:2454] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:45.505156 32553 raft_consensus.cc:2454] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:45.505405 32621 leader_election.cc:302] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:45.505566 4829 raft_consensus.cc:2781] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:45.505620 4829 raft_consensus.cc:482] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:45.505645 4829 raft_consensus.cc:3037] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.506440 4829 raft_consensus.cc:504] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:45.506776 4829 leader_election.cc:288] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Requested vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:45.507026 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6af3f1cff644f4091e530d81e958ffa" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:45.507143 32415 raft_consensus.cc:3037] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.507110 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6af3f1cff644f4091e530d81e958ffa" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:45.507216 32553 raft_consensus.cc:3037] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.508117 32553 raft_consensus.cc:2454] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:45.508117 32415 raft_consensus.cc:2454] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:45.508505 32620 leader_election.cc:302] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:45.508687 4829 raft_consensus.cc:2781] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:45.508770 4829 raft_consensus.cc:686] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Becoming Leader. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
I20240501 14:09:45.508904 4829 consensus_queue.cc:237] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:45.509765 5037 ts_tablet_manager.cc:902] T 15c98173a71d4f73b8f6af24ad01d7c4 P a91029b90bf9415a89a2f70736759ad3: Initiating tablet copy from peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:45.510277 5037 tablet_copy_client.cc:308] T 15c98173a71d4f73b8f6af24ad01d7c4 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.129:41387
I20240501 14:09:45.510699 30035 tablet_copy_service.cc:140] P 93f2518fbadf4f9991cda93afacb363e: Received BeginTabletCopySession request for tablet 15c98173a71d4f73b8f6af24ad01d7c4 from peer a91029b90bf9415a89a2f70736759ad3 ({username='slave'} at 127.22.252.133:60917)
I20240501 14:09:45.510795 30035 tablet_copy_service.cc:161] P 93f2518fbadf4f9991cda93afacb363e: Beginning new tablet copy session on tablet 15c98173a71d4f73b8f6af24ad01d7c4 from peer a91029b90bf9415a89a2f70736759ad3 at {username='slave'} at 127.22.252.133:60917: session id = a91029b90bf9415a89a2f70736759ad3-15c98173a71d4f73b8f6af24ad01d7c4
I20240501 14:09:45.511138 29871 catalog_manager.cc:5484] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db935501e9104de5b15c8780681d6c38 (127.22.252.135). New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:45.511596 30035 tablet_copy_source_session.cc:205] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:45.512413 5037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15c98173a71d4f73b8f6af24ad01d7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.514358 5037 tablet_copy_client.cc:783] T 15c98173a71d4f73b8f6af24ad01d7c4 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:45.514488 5037 tablet_copy_client.cc:647] T 15c98173a71d4f73b8f6af24ad01d7c4 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:45.515515 5037 tablet_copy_client.cc:523] T 15c98173a71d4f73b8f6af24ad01d7c4 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:45.516510 5037 tablet_bootstrap.cc:492] T 15c98173a71d4f73b8f6af24ad01d7c4 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:45.542141 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:45.547581 30124 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:45.551091 5037 tablet_bootstrap.cc:492] T 15c98173a71d4f73b8f6af24ad01d7c4 P a91029b90bf9415a89a2f70736759ad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:45.551506 5037 tablet_bootstrap.cc:492] T 15c98173a71d4f73b8f6af24ad01d7c4 P a91029b90bf9415a89a2f70736759ad3: Bootstrap complete.
I20240501 14:09:45.551743 5037 ts_tablet_manager.cc:1367] T 15c98173a71d4f73b8f6af24ad01d7c4 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.035s user 0.028s sys 0.000s
I20240501 14:09:45.552263 5037 raft_consensus.cc:348] T 15c98173a71d4f73b8f6af24ad01d7c4 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:45.552466 5037 raft_consensus.cc:729] T 15c98173a71d4f73b8f6af24ad01d7c4 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.552639 5037 consensus_queue.cc:260] T 15c98173a71d4f73b8f6af24ad01d7c4 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:45.553098 5037 ts_tablet_manager.cc:1398] T 15c98173a71d4f73b8f6af24ad01d7c4 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.553597 30035 tablet_copy_service.cc:342] P 93f2518fbadf4f9991cda93afacb363e: Request end of tablet copy session a91029b90bf9415a89a2f70736759ad3-15c98173a71d4f73b8f6af24ad01d7c4 received from {username='slave'} at 127.22.252.133:60917
I20240501 14:09:45.553678 30035 tablet_copy_service.cc:433] P 93f2518fbadf4f9991cda93afacb363e: ending tablet copy session a91029b90bf9415a89a2f70736759ad3-15c98173a71d4f73b8f6af24ad01d7c4 on tablet 15c98173a71d4f73b8f6af24ad01d7c4 with peer a91029b90bf9415a89a2f70736759ad3
I20240501 14:09:45.554920 327 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:45.555258 32400 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:45.561381 5054 ts_tablet_manager.cc:902] T 2c5c502325fb475f8f050ccad34feb2a P d35c16dcb506426993811bb2519d7db3: Initiating tablet copy from peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:45.561992 5054 tablet_copy_client.cc:308] T 2c5c502325fb475f8f050ccad34feb2a P d35c16dcb506426993811bb2519d7db3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.133:46265
I20240501 14:09:45.562036 32534 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:45.562577 32443 tablet_copy_service.cc:140] P a91029b90bf9415a89a2f70736759ad3: Received BeginTabletCopySession request for tablet 2c5c502325fb475f8f050ccad34feb2a from peer d35c16dcb506426993811bb2519d7db3 ({username='slave'} at 127.22.252.130:47505)
I20240501 14:09:45.562662 32443 tablet_copy_service.cc:161] P a91029b90bf9415a89a2f70736759ad3: Beginning new tablet copy session on tablet 2c5c502325fb475f8f050ccad34feb2a from peer d35c16dcb506426993811bb2519d7db3 at {username='slave'} at 127.22.252.130:47505: session id = d35c16dcb506426993811bb2519d7db3-2c5c502325fb475f8f050ccad34feb2a
I20240501 14:09:45.562868 30388 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:45.562971 32664 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:45.563607 32443 tablet_copy_source_session.cc:205] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:45.564610 5054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5c502325fb475f8f050ccad34feb2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.577850 5054 tablet_copy_client.cc:783] T 2c5c502325fb475f8f050ccad34feb2a P d35c16dcb506426993811bb2519d7db3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:45.578050 5054 tablet_copy_client.cc:647] T 2c5c502325fb475f8f050ccad34feb2a P d35c16dcb506426993811bb2519d7db3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:45.580425 5054 tablet_copy_client.cc:523] T 2c5c502325fb475f8f050ccad34feb2a P d35c16dcb506426993811bb2519d7db3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:45.581722 5054 tablet_bootstrap.cc:492] T 2c5c502325fb475f8f050ccad34feb2a P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:45.615710 29995 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:45.619506 5054 tablet_bootstrap.cc:492] T 2c5c502325fb475f8f050ccad34feb2a P d35c16dcb506426993811bb2519d7db3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:45.621270 5054 tablet_bootstrap.cc:492] T 2c5c502325fb475f8f050ccad34feb2a P d35c16dcb506426993811bb2519d7db3: Bootstrap complete.
I20240501 14:09:45.621450 5054 ts_tablet_manager.cc:1367] T 2c5c502325fb475f8f050ccad34feb2a P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20240501 14:09:45.621860 5054 raft_consensus.cc:348] T 2c5c502325fb475f8f050ccad34feb2a P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:45.627614 5054 raft_consensus.cc:729] T 2c5c502325fb475f8f050ccad34feb2a P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.627826 5054 consensus_queue.cc:260] T 2c5c502325fb475f8f050ccad34feb2a P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:45.628175 5054 ts_tablet_manager.cc:1398] T 2c5c502325fb475f8f050ccad34feb2a P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:45.628837 32443 tablet_copy_service.cc:342] P a91029b90bf9415a89a2f70736759ad3: Request end of tablet copy session d35c16dcb506426993811bb2519d7db3-2c5c502325fb475f8f050ccad34feb2a received from {username='slave'} at 127.22.252.130:47505
I20240501 14:09:45.628921 32443 tablet_copy_service.cc:433] P a91029b90bf9415a89a2f70736759ad3: ending tablet copy session d35c16dcb506426993811bb2519d7db3-2c5c502325fb475f8f050ccad34feb2a on tablet 2c5c502325fb475f8f050ccad34feb2a with peer d35c16dcb506426993811bb2519d7db3
I20240501 14:09:45.640652 29871 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:45.645489 32415 raft_consensus.cc:1261] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:45.645538 32553 raft_consensus.cc:1261] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:45.645892 4830 consensus_queue.cc:1035] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.646109 4829 consensus_queue.cc:1035] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.649547 29871 catalog_manager.cc:5576] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497) does not have the latest schema for tablet e6af3f1cff644f4091e530d81e958ffa (table rebalancer_dynamic_table [id=0dba7d13b364430ab8806c35258df29d]). Expected version 1 got 0
I20240501 14:09:45.650956 5071 tablet.cc:1722] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:45.651470 5071 tablet.cc:1620] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:45.651875 29871 catalog_manager.cc:5576] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265) does not have the latest schema for tablet e6af3f1cff644f4091e530d81e958ffa (table rebalancer_dynamic_table [id=0dba7d13b364430ab8806c35258df29d]). Expected version 1 got 0
I20240501 14:09:45.653541 29871 catalog_manager.cc:5576] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497) does not have the latest schema for tablet e6af3f1cff644f4091e530d81e958ffa (table rebalancer_dynamic_table [id=0dba7d13b364430ab8806c35258df29d]). Expected version 1 got 0
I20240501 14:09:45.655984 29871 catalog_manager.cc:5576] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265) does not have the latest schema for tablet e6af3f1cff644f4091e530d81e958ffa (table rebalancer_dynamic_table [id=0dba7d13b364430ab8806c35258df29d]). Expected version 1 got 0
I20240501 14:09:45.656275 5072 tablet.cc:1722] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:45.656720 5072 tablet.cc:1620] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:45.657555 29869 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=0dba7d13b364430ab8806c35258df29d] alter complete (version 1)
I20240501 14:09:45.658304 5079 tablet.cc:1722] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:45.658640 5079 tablet.cc:1620] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:45.713165 4371 consensus_queue.cc:1035] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.723151 2864 consensus_queue.cc:1035] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.731003 4468 consensus_queue.cc:1035] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.736518 4855 consensus_queue.cc:1035] T aa8ba266a8d0406bb93d6c578fe6c7cb P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.747902 4094 consensus_queue.cc:1035] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.752740 4094 consensus_queue.cc:1035] T 609a7f5fccdf4a8a9f854d9007412d33 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.755584 32534 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:45.757053 30388 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:45.767230 30124 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:45.767431 29866 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:45.768843 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:45.770502 327 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:45.784072 5071 tablet.cc:1722] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:45.784457 5071 tablet.cc:1620] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:45.786391 29995 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:45.788857 5072 tablet.cc:1722] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:45.789182 5072 tablet.cc:1620] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:45.789264 5079 tablet.cc:1722] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:45.789561 5079 tablet.cc:1620] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:45.793599 29865 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=0dba7d13b364430ab8806c35258df29d] alter complete (version 2)
I20240501 14:09:45.798559 32664 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:45.806313 30406 raft_consensus.cc:1203] T 7a6a29261186429cbc05b1819999f986 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:45.814131 32553 raft_consensus.cc:1203] T c7df6b8d76714eebbeacc85f403d3405 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:45.820829 32400 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:45.842114 29866 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:45.852351 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb9687045d44efa888a69c7acde1105 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c0df7ced31ed4653bf4f71f2bb576ed9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.852433 32532 tablet_service.cc:1467] Processing CreateTablet for tablet 7e337eb723974834b07aac5eacf73232 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c0df7ced31ed4653bf4f71f2bb576ed9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.852433 327 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb9687045d44efa888a69c7acde1105 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c0df7ced31ed4653bf4f71f2bb576ed9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.852658 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb9687045d44efa888a69c7acde1105. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.852710 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb9687045d44efa888a69c7acde1105. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.853098 32400 tablet_service.cc:1467] Processing CreateTablet for tablet eb828a6a5cd240f5acb591e7eae5e1a8 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c0df7ced31ed4653bf4f71f2bb576ed9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.853359 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb828a6a5cd240f5acb591e7eae5e1a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.854928 329 tablet_service.cc:1467] Processing CreateTablet for tablet 7e337eb723974834b07aac5eacf73232 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c0df7ced31ed4653bf4f71f2bb576ed9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.855118 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e337eb723974834b07aac5eacf73232. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.855259 29995 tablet_service.cc:1467] Processing CreateTablet for tablet eb828a6a5cd240f5acb591e7eae5e1a8 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c0df7ced31ed4653bf4f71f2bb576ed9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.855533 29995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb828a6a5cd240f5acb591e7eae5e1a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.855875 5136 tablet_bootstrap.cc:492] T eb828a6a5cd240f5acb591e7eae5e1a8 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:45.856025 32532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e337eb723974834b07aac5eacf73232. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.852351 32533 tablet_service.cc:1467] Processing CreateTablet for tablet eb828a6a5cd240f5acb591e7eae5e1a8 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c0df7ced31ed4653bf4f71f2bb576ed9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.856858 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb828a6a5cd240f5acb591e7eae5e1a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.857113 5136 tablet_bootstrap.cc:654] T eb828a6a5cd240f5acb591e7eae5e1a8 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.857862 5138 tablet_bootstrap.cc:492] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:45.858970 5138 tablet_bootstrap.cc:654] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.859598 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb9687045d44efa888a69c7acde1105 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c0df7ced31ed4653bf4f71f2bb576ed9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.859656 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 7e337eb723974834b07aac5eacf73232 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c0df7ced31ed4653bf4f71f2bb576ed9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.859848 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb9687045d44efa888a69c7acde1105. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.859974 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e337eb723974834b07aac5eacf73232. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.861565 5135 tablet_bootstrap.cc:492] T 8bb9687045d44efa888a69c7acde1105 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:45.861642 5134 tablet_bootstrap.cc:492] T 8bb9687045d44efa888a69c7acde1105 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:45.864005 5134 tablet_bootstrap.cc:654] T 8bb9687045d44efa888a69c7acde1105 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.864276 5139 tablet_bootstrap.cc:492] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:45.865458 5139 tablet_bootstrap.cc:654] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.867623 5135 tablet_bootstrap.cc:654] T 8bb9687045d44efa888a69c7acde1105 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.868705 5139 tablet_bootstrap.cc:492] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:45.869019 5139 ts_tablet_manager.cc:1367] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:45.869510 5139 raft_consensus.cc:348] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:45.869701 5139 raft_consensus.cc:374] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.869774 5139 raft_consensus.cc:729] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.869977 5135 tablet_bootstrap.cc:492] T 8bb9687045d44efa888a69c7acde1105 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:45.870134 5135 ts_tablet_manager.cc:1367] T 8bb9687045d44efa888a69c7acde1105 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:45.870303 5136 tablet_bootstrap.cc:492] T eb828a6a5cd240f5acb591e7eae5e1a8 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:45.870393 5136 ts_tablet_manager.cc:1367] T eb828a6a5cd240f5acb591e7eae5e1a8 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 14:09:45.870645 5135 raft_consensus.cc:348] T 8bb9687045d44efa888a69c7acde1105 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:45.870740 5135 raft_consensus.cc:374] T 8bb9687045d44efa888a69c7acde1105 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.870770 5135 raft_consensus.cc:729] T 8bb9687045d44efa888a69c7acde1105 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.870847 5136 raft_consensus.cc:348] T eb828a6a5cd240f5acb591e7eae5e1a8 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:45.870945 5136 raft_consensus.cc:374] T eb828a6a5cd240f5acb591e7eae5e1a8 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.870914 5135 consensus_queue.cc:260] T 8bb9687045d44efa888a69c7acde1105 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:45.871006 5136 raft_consensus.cc:729] T eb828a6a5cd240f5acb591e7eae5e1a8 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.871011 5139 consensus_queue.cc:260] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:45.871126 5136 consensus_queue.cc:260] T eb828a6a5cd240f5acb591e7eae5e1a8 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:45.871301 5139 ts_tablet_manager.cc:1398] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:45.871472 5139 tablet_bootstrap.cc:492] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:45.871737 5136 ts_tablet_manager.cc:1398] T eb828a6a5cd240f5acb591e7eae5e1a8 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.872331 5139 tablet_bootstrap.cc:654] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.873291 5138 tablet_bootstrap.cc:492] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:45.873379 5138 ts_tablet_manager.cc:1367] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:09:45.873901 5138 raft_consensus.cc:348] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:45.874007 5138 raft_consensus.cc:374] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.874042 5138 raft_consensus.cc:729] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.874146 5138 consensus_queue.cc:260] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:45.874424 5138 ts_tablet_manager.cc:1398] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.874722 5134 tablet_bootstrap.cc:492] T 8bb9687045d44efa888a69c7acde1105 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:45.874806 5134 ts_tablet_manager.cc:1367] T 8bb9687045d44efa888a69c7acde1105 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:09:45.875264 5134 raft_consensus.cc:348] T 8bb9687045d44efa888a69c7acde1105 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:45.875375 5134 raft_consensus.cc:374] T 8bb9687045d44efa888a69c7acde1105 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.875411 5134 raft_consensus.cc:729] T 8bb9687045d44efa888a69c7acde1105 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.875535 5134 consensus_queue.cc:260] T 8bb9687045d44efa888a69c7acde1105 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:45.875906 5134 ts_tablet_manager.cc:1398] T 8bb9687045d44efa888a69c7acde1105 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.876128 5134 tablet_bootstrap.cc:492] T 7e337eb723974834b07aac5eacf73232 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:45.876358 5139 tablet_bootstrap.cc:492] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:45.876432 5139 ts_tablet_manager.cc:1367] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:09:45.876844 5135 ts_tablet_manager.cc:1398] T 8bb9687045d44efa888a69c7acde1105 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:45.876981 5139 raft_consensus.cc:348] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:45.877071 5139 raft_consensus.cc:374] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.877100 5139 raft_consensus.cc:729] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.877202 5139 consensus_queue.cc:260] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:45.877447 5139 ts_tablet_manager.cc:1398] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.878141 5134 tablet_bootstrap.cc:654] T 7e337eb723974834b07aac5eacf73232 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.880832 5134 tablet_bootstrap.cc:492] T 7e337eb723974834b07aac5eacf73232 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:45.880923 5134 ts_tablet_manager.cc:1367] T 7e337eb723974834b07aac5eacf73232 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:45.881457 5134 raft_consensus.cc:348] T 7e337eb723974834b07aac5eacf73232 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:45.881582 5134 raft_consensus.cc:374] T 7e337eb723974834b07aac5eacf73232 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.881666 5134 raft_consensus.cc:729] T 7e337eb723974834b07aac5eacf73232 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.881886 5135 tablet_bootstrap.cc:492] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:45.881915 5134 consensus_queue.cc:260] T 7e337eb723974834b07aac5eacf73232 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:45.882272 5134 ts_tablet_manager.cc:1398] T 7e337eb723974834b07aac5eacf73232 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.882493 5134 tablet_bootstrap.cc:492] T eb828a6a5cd240f5acb591e7eae5e1a8 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:45.882928 5135 tablet_bootstrap.cc:654] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.883620 5134 tablet_bootstrap.cc:654] T eb828a6a5cd240f5acb591e7eae5e1a8 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.886844 5134 tablet_bootstrap.cc:492] T eb828a6a5cd240f5acb591e7eae5e1a8 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:45.886940 5134 ts_tablet_manager.cc:1367] T eb828a6a5cd240f5acb591e7eae5e1a8 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:45.887383 5134 raft_consensus.cc:348] T eb828a6a5cd240f5acb591e7eae5e1a8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:45.887529 5134 raft_consensus.cc:374] T eb828a6a5cd240f5acb591e7eae5e1a8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.887604 5134 raft_consensus.cc:729] T eb828a6a5cd240f5acb591e7eae5e1a8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.887869 5134 consensus_queue.cc:260] T eb828a6a5cd240f5acb591e7eae5e1a8 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:45.888312 5134 ts_tablet_manager.cc:1398] T eb828a6a5cd240f5acb591e7eae5e1a8 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.891151 5135 tablet_bootstrap.cc:492] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:45.891249 5135 ts_tablet_manager.cc:1367] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:45.891662 5135 raft_consensus.cc:348] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:45.891757 5135 raft_consensus.cc:374] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.891790 5135 raft_consensus.cc:729] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.891933 5135 consensus_queue.cc:260] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:45.892203 5135 ts_tablet_manager.cc:1398] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.896342 29872 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:45.903729 5071 tablet.cc:1722] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:45.904460 5079 tablet.cc:1722] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:45.904491 5072 tablet.cc:1722] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:45.910488 29866 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=0dba7d13b364430ab8806c35258df29d] alter complete (version 3)
I20240501 14:09:45.980499 4830 raft_consensus.cc:482] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:45.980631 4830 raft_consensus.cc:504] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:45.981078 4830 leader_election.cc:288] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:45.981832 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e337eb723974834b07aac5eacf73232" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:45.981968 349 raft_consensus.cc:2454] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:45.982257 32620 leader_election.cc:302] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:45.982353 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e337eb723974834b07aac5eacf73232" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:45.982417 4830 raft_consensus.cc:2781] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:45.982457 4830 raft_consensus.cc:482] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:45.982496 4830 raft_consensus.cc:3037] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.982481 32553 raft_consensus.cc:2454] T 7e337eb723974834b07aac5eacf73232 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:45.983676 4830 raft_consensus.cc:504] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:45.983975 4830 leader_election.cc:288] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:45.984380 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e337eb723974834b07aac5eacf73232" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:45.984448 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e337eb723974834b07aac5eacf73232" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:45.984483 32553 raft_consensus.cc:3037] T 7e337eb723974834b07aac5eacf73232 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.984539 349 raft_consensus.cc:3037] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.985427 349 raft_consensus.cc:2454] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:45.985427 32553 raft_consensus.cc:2454] T 7e337eb723974834b07aac5eacf73232 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:45.985698 32620 leader_election.cc:302] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:45.987129 4830 raft_consensus.cc:2781] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:45.987214 4830 raft_consensus.cc:686] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Becoming Leader. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
I20240501 14:09:45.987344 4830 consensus_queue.cc:237] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:45.990128 29868 catalog_manager.cc:5484] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db935501e9104de5b15c8780681d6c38 (127.22.252.135). New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:45.996663 349 raft_consensus.cc:1203] T e122adbba7c749cd93c53fa07a8a8b84 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:45.997948 5130 raft_consensus.cc:482] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:45.998064 5130 raft_consensus.cc:504] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:45.998525 5130 leader_election.cc:288] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:45.999272 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb828a6a5cd240f5acb591e7eae5e1a8" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:45.999435 32415 raft_consensus.cc:2454] T eb828a6a5cd240f5acb591e7eae5e1a8 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:46.000314 29950 leader_election.cc:302] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:46.000532 5130 raft_consensus.cc:2781] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:46.000591 5130 raft_consensus.cc:482] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:46.000635 5130 raft_consensus.cc:3037] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.001780 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb828a6a5cd240f5acb591e7eae5e1a8" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:46.001909 32553 raft_consensus.cc:2454] T eb828a6a5cd240f5acb591e7eae5e1a8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:46.001904 5130 raft_consensus.cc:504] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:46.002233 5130 leader_election.cc:288] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:46.002791 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb828a6a5cd240f5acb591e7eae5e1a8" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:46.002913 32553 raft_consensus.cc:3037] T eb828a6a5cd240f5acb591e7eae5e1a8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.003211 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb828a6a5cd240f5acb591e7eae5e1a8" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:46.003337 32415 raft_consensus.cc:3037] T eb828a6a5cd240f5acb591e7eae5e1a8 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.003938 32553 raft_consensus.cc:2454] T eb828a6a5cd240f5acb591e7eae5e1a8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:46.004599 32416 raft_consensus.cc:1203] T 15c98173a71d4f73b8f6af24ad01d7c4 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:46.005218 29951 leader_election.cc:302] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:46.005512 32415 raft_consensus.cc:2454] T eb828a6a5cd240f5acb591e7eae5e1a8 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:46.006142 5130 raft_consensus.cc:2781] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:46.006244 5130 raft_consensus.cc:686] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:46.006384 5130 consensus_queue.cc:237] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:46.008498 29868 catalog_manager.cc:5484] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:46.013239 29868 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:46.013350 29868 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:46.014950 29868 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet e6af3f1cff644f4091e530d81e958ffa
I20240501 14:09:46.015406 32400 tablet_service.cc:1514] Processing DeleteTablet for tablet e6af3f1cff644f4091e530d81e958ffa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:46 UTC) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:46.015575 32664 tablet_service.cc:1514] Processing DeleteTablet for tablet e6af3f1cff644f4091e530d81e958ffa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:46 UTC) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:46.015812 32533 tablet_service.cc:1514] Processing DeleteTablet for tablet e6af3f1cff644f4091e530d81e958ffa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:46 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:46.015872 5191 tablet_replica.cc:330] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:46.016115 5193 tablet_replica.cc:330] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:46.016185 5192 tablet_replica.cc:330] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:46.016330 5193 raft_consensus.cc:2229] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:46.016482 5192 raft_consensus.cc:2229] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:46.016512 5193 raft_consensus.cc:2258] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:46.016780 5192 raft_consensus.cc:2258] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:46.017131 5193 ts_tablet_manager.cc:1875] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:46.018023 5191 raft_consensus.cc:2229] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:46.018211 5191 raft_consensus.cc:2258] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:46.018640 5192 ts_tablet_manager.cc:1875] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:46.019042 5191 ts_tablet_manager.cc:1875] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:46.019883 5193 ts_tablet_manager.cc:1888] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:46.020018 5193 log.cc:1196] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/e6af3f1cff644f4091e530d81e958ffa
I20240501 14:09:46.020401 5193 ts_tablet_manager.cc:1909] T e6af3f1cff644f4091e530d81e958ffa P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:46.021071 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet e6af3f1cff644f4091e530d81e958ffa (table rebalancer_dynamic_table_new_name [id=0dba7d13b364430ab8806c35258df29d]) successfully deleted
I20240501 14:09:46.021858 5192 ts_tablet_manager.cc:1888] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:46.021858 5191 ts_tablet_manager.cc:1888] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:46.021935 5191 log.cc:1196] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/e6af3f1cff644f4091e530d81e958ffa
I20240501 14:09:46.021935 5192 log.cc:1196] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/e6af3f1cff644f4091e530d81e958ffa
I20240501 14:09:46.022231 5192 ts_tablet_manager.cc:1909] T e6af3f1cff644f4091e530d81e958ffa P db935501e9104de5b15c8780681d6c38: Deleting consensus metadata
I20240501 14:09:46.022275 5191 ts_tablet_manager.cc:1909] T e6af3f1cff644f4091e530d81e958ffa P a91029b90bf9415a89a2f70736759ad3: Deleting consensus metadata
I20240501 14:09:46.022977 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet e6af3f1cff644f4091e530d81e958ffa (table rebalancer_dynamic_table_new_name [id=0dba7d13b364430ab8806c35258df29d]) successfully deleted
I20240501 14:09:46.023326 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet e6af3f1cff644f4091e530d81e958ffa (table rebalancer_dynamic_table_new_name [id=0dba7d13b364430ab8806c35258df29d]) successfully deleted
I20240501 14:09:46.116688 29868 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:46.121802 32533 tablet_service.cc:1467] Processing CreateTablet for tablet 6e93e121a8854902b54f06c983ba5b9c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6fab5ac330974a39be5a7c7e5774950a]), partition=
I20240501 14:09:46.121906 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 6e93e121a8854902b54f06c983ba5b9c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6fab5ac330974a39be5a7c7e5774950a]), partition=
I20240501 14:09:46.122092 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e93e121a8854902b54f06c983ba5b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.122077 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 6e93e121a8854902b54f06c983ba5b9c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6fab5ac330974a39be5a7c7e5774950a]), partition=
I20240501 14:09:46.122161 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e93e121a8854902b54f06c983ba5b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.122291 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e93e121a8854902b54f06c983ba5b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.123944 5134 tablet_bootstrap.cc:492] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:46.124430 5136 tablet_bootstrap.cc:492] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:46.124430 5139 tablet_bootstrap.cc:492] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:46.125077 5134 tablet_bootstrap.cc:654] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.125253 5139 tablet_bootstrap.cc:654] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.125254 5136 tablet_bootstrap.cc:654] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.126080 5134 tablet_bootstrap.cc:492] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:46.126163 5134 ts_tablet_manager.cc:1367] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:46.126631 5139 tablet_bootstrap.cc:492] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:46.126706 5139 ts_tablet_manager.cc:1367] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:46.126636 5134 raft_consensus.cc:348] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:46.126752 5134 raft_consensus.cc:374] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.126791 5134 raft_consensus.cc:729] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.126910 5134 consensus_queue.cc:260] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:46.127123 5139 raft_consensus.cc:348] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:46.127213 5134 ts_tablet_manager.cc:1398] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:46.127226 5139 raft_consensus.cc:374] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.127259 5139 raft_consensus.cc:729] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.127365 5139 consensus_queue.cc:260] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:46.127629 5139 ts_tablet_manager.cc:1398] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:46.128453 5136 tablet_bootstrap.cc:492] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:46.128545 5136 ts_tablet_manager.cc:1367] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:46.128886 5136 raft_consensus.cc:348] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:46.128968 5136 raft_consensus.cc:374] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.129000 5136 raft_consensus.cc:729] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.129145 5136 consensus_queue.cc:260] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:46.129407 5136 ts_tablet_manager.cc:1398] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:46.137593 4829 raft_consensus.cc:482] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:46.137673 4829 raft_consensus.cc:504] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:46.138067 4829 leader_election.cc:288] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:46.138352 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e93e121a8854902b54f06c983ba5b9c" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:46.138360 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e93e121a8854902b54f06c983ba5b9c" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:46.138478 32554 raft_consensus.cc:2454] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:46.138475 32415 raft_consensus.cc:2454] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:46.138748 32621 leader_election.cc:302] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:46.138918 4829 raft_consensus.cc:2781] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:46.138991 4829 raft_consensus.cc:482] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:46.139039 4829 raft_consensus.cc:3037] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.139719 30146 raft_consensus.cc:1203] T 2c5c502325fb475f8f050ccad34feb2a P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:46.139874 4829 raft_consensus.cc:504] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:46.140179 4829 leader_election.cc:288] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:46.140897 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e93e121a8854902b54f06c983ba5b9c" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:46.141005 32554 raft_consensus.cc:3037] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.141050 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e93e121a8854902b54f06c983ba5b9c" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:46.141132 32415 raft_consensus.cc:3037] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.141988 32554 raft_consensus.cc:2454] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:46.142058 32415 raft_consensus.cc:2454] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:46.142450 32621 leader_election.cc:302] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:46.142601 4829 raft_consensus.cc:2781] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:46.142712 4829 raft_consensus.cc:686] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Becoming Leader. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
I20240501 14:09:46.142849 4829 consensus_queue.cc:237] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:46.144914 29872 catalog_manager.cc:5484] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db935501e9104de5b15c8780681d6c38 (127.22.252.135). New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:46.228543 4829 raft_consensus.cc:482] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:46.228672 4829 raft_consensus.cc:504] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:46.229058 4829 leader_election.cc:288] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:46.229471 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb9687045d44efa888a69c7acde1105" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:46.229473 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb9687045d44efa888a69c7acde1105" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:46.229602 349 raft_consensus.cc:2454] T 8bb9687045d44efa888a69c7acde1105 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:46.229605 32554 raft_consensus.cc:2454] T 8bb9687045d44efa888a69c7acde1105 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:46.229868 32621 leader_election.cc:302] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:46.230046 4829 raft_consensus.cc:2781] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:46.230111 4829 raft_consensus.cc:482] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:46.230139 4829 raft_consensus.cc:3037] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.230906 4829 raft_consensus.cc:504] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:46.231200 4829 leader_election.cc:288] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:46.231519 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb9687045d44efa888a69c7acde1105" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:46.231520 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb9687045d44efa888a69c7acde1105" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:46.231627 32554 raft_consensus.cc:3037] T 8bb9687045d44efa888a69c7acde1105 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.231627 349 raft_consensus.cc:3037] T 8bb9687045d44efa888a69c7acde1105 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.232606 32554 raft_consensus.cc:2454] T 8bb9687045d44efa888a69c7acde1105 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:46.232606 349 raft_consensus.cc:2454] T 8bb9687045d44efa888a69c7acde1105 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:46.232892 32620 leader_election.cc:302] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:46.233124 4829 raft_consensus.cc:2781] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:46.233214 4829 raft_consensus.cc:686] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Becoming Leader. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
I20240501 14:09:46.233350 4829 consensus_queue.cc:237] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:46.235195 29872 catalog_manager.cc:5484] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db935501e9104de5b15c8780681d6c38 (127.22.252.135). New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:46.252811 32533 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:46.254527 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:46.255514 29872 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:46.256204 29995 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:46.258998 30388 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:46.262320 32415 raft_consensus.cc:1261] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:46.262799 4829 consensus_queue.cc:1035] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.265970 29868 catalog_manager.cc:5576] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265) does not have the latest schema for tablet 6e93e121a8854902b54f06c983ba5b9c (table rebalancer_dynamic_table [id=6fab5ac330974a39be5a7c7e5774950a]). Expected version 1 got 0
I20240501 14:09:46.267664 32554 raft_consensus.cc:1261] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:46.268007 4829 consensus_queue.cc:1035] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.270076 329 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:46.272384 30124 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:46.274718 29868 catalog_manager.cc:5576] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497) does not have the latest schema for tablet 6e93e121a8854902b54f06c983ba5b9c (table rebalancer_dynamic_table [id=6fab5ac330974a39be5a7c7e5774950a]). Expected version 1 got 0
I20240501 14:09:46.278530 32400 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:46.287189 5071 tablet.cc:1722] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:46.287570 5071 tablet.cc:1620] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:46.288669 5072 tablet.cc:1722] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:46.288930 5072 tablet.cc:1620] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:46.293071 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=6fab5ac330974a39be5a7c7e5774950a] alter complete (version 1)
I20240501 14:09:46.301072 5079 tablet.cc:1722] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:46.301389 5079 tablet.cc:1620] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:46.314564 32664 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:46.355595 32415 consensus_queue.cc:237] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:46.356786 30146 raft_consensus.cc:1261] T 575e8cde863540b09a52fe590cbf2bfc P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:46.356812 32554 raft_consensus.cc:1261] T 575e8cde863540b09a52fe590cbf2bfc P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:46.357141 4825 consensus_queue.cc:1035] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:46.357345 4874 consensus_queue.cc:1035] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:46.359055 5230 raft_consensus.cc:2932] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:46.359429 30146 raft_consensus.cc:2932] T 575e8cde863540b09a52fe590cbf2bfc P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:46.359548 32554 raft_consensus.cc:2932] T 575e8cde863540b09a52fe590cbf2bfc P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:46.362097 29868 catalog_manager.cc:5484] T 575e8cde863540b09a52fe590cbf2bfc P d35c16dcb506426993811bb2519d7db3 reported cstate change: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:46.365069 32664 tablet_service.cc:1514] Processing DeleteTablet for tablet 575e8cde863540b09a52fe590cbf2bfc with delete_type TABLET_DATA_TOMBSTONED (TS db935501e9104de5b15c8780681d6c38 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:46.365312 5192 tablet_replica.cc:330] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:46.365458 5192 raft_consensus.cc:2229] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:46.365581 5192 raft_consensus.cc:2258] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:46.365959 5192 ts_tablet_manager.cc:1875] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:46.367970 5192 ts_tablet_manager.cc:1888] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:09:46.368032 5192 log.cc:1196] T 575e8cde863540b09a52fe590cbf2bfc P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/575e8cde863540b09a52fe590cbf2bfc
I20240501 14:09:46.368553 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet 575e8cde863540b09a52fe590cbf2bfc (table rebalancer_extra_table_0 [id=25cba3c30d2b44599355b7317baedb1c]) successfully deleted
I20240501 14:09:46.397001 29872 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:46.403424 5071 tablet.cc:1722] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:46.403764 5071 tablet.cc:1620] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:46.403908 5072 tablet.cc:1722] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:46.403947 5079 tablet.cc:1722] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:46.404165 5072 tablet.cc:1620] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:46.404719 5079 tablet.cc:1620] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:46.408931 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=6fab5ac330974a39be5a7c7e5774950a] alter complete (version 2)
I20240501 14:09:46.464599 4852 consensus_queue.cc:1035] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.468297 32533 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:46.474498 5130 consensus_queue.cc:1035] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.478093 32400 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:46.482862 30388 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:46.483275 32664 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:46.483565 29995 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:46.483932 329 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:46.492599 30124 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:46.503906 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:46.507468 4830 consensus_queue.cc:1035] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.510504 5070 consensus_queue.cc:1035] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.513336 29868 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:46.519155 5071 tablet.cc:1722] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:46.520016 5072 tablet.cc:1722] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:46.529163 5079 tablet.cc:1722] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:46.533550 29866 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=6fab5ac330974a39be5a7c7e5774950a] alter complete (version 3)
I20240501 14:09:46.568673 32684 consensus_queue.cc:237] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:46.569844 349 raft_consensus.cc:1261] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:46.569850 30277 raft_consensus.cc:1261] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:46.570377 5070 consensus_queue.cc:1035] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:46.570580 4830 consensus_queue.cc:1035] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:46.572221 4829 raft_consensus.cc:2932] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:46.572630 349 raft_consensus.cc:2932] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:46.572628 30277 raft_consensus.cc:2932] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:46.575182 29867 catalog_manager.cc:5484] T 0ca8c2fa60c543dab6f30acce1958573 P db935501e9104de5b15c8780681d6c38 reported cstate change: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:46.577873 32400 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ca8c2fa60c543dab6f30acce1958573 with delete_type TABLET_DATA_TOMBSTONED (TS a91029b90bf9415a89a2f70736759ad3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:46.578231 5307 tablet_replica.cc:330] T 0ca8c2fa60c543dab6f30acce1958573 P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:46.578415 5307 raft_consensus.cc:2229] T 0ca8c2fa60c543dab6f30acce1958573 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:46.578534 5307 raft_consensus.cc:2258] T 0ca8c2fa60c543dab6f30acce1958573 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:46.578927 5307 ts_tablet_manager.cc:1875] T 0ca8c2fa60c543dab6f30acce1958573 P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:46.581110 5307 ts_tablet_manager.cc:1888] T 0ca8c2fa60c543dab6f30acce1958573 P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:09:46.581246 5307 log.cc:1196] T 0ca8c2fa60c543dab6f30acce1958573 P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/0ca8c2fa60c543dab6f30acce1958573
I20240501 14:09:46.581787 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 0ca8c2fa60c543dab6f30acce1958573 (table rebalancer_extra_table_4 [id=ec60e0d43df9403baaddddba692166ca]) successfully deleted
I20240501 14:09:46.644372 29872 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:46.644510 29872 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:46.646654 29872 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 6e93e121a8854902b54f06c983ba5b9c
I20240501 14:09:46.647166 32533 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e93e121a8854902b54f06c983ba5b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:46 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:46.647298 32400 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e93e121a8854902b54f06c983ba5b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:46 UTC) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:46.647480 5307 tablet_replica.cc:330] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:46.647622 5307 raft_consensus.cc:2229] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:46.647689 32664 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e93e121a8854902b54f06c983ba5b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:46 UTC) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:46.647756 5307 raft_consensus.cc:2258] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:46.647898 5192 tablet_replica.cc:330] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:46.648048 5192 raft_consensus.cc:2229] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:46.648304 5192 raft_consensus.cc:2258] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:46.648974 5192 ts_tablet_manager.cc:1875] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:46.650292 5307 ts_tablet_manager.cc:1875] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:46.651266 5192 ts_tablet_manager.cc:1888] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:46.652336 5307 ts_tablet_manager.cc:1888] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:46.652401 5307 log.cc:1196] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/6e93e121a8854902b54f06c983ba5b9c
I20240501 14:09:46.652659 5307 ts_tablet_manager.cc:1909] T 6e93e121a8854902b54f06c983ba5b9c P a91029b90bf9415a89a2f70736759ad3: Deleting consensus metadata
I20240501 14:09:46.652863 5192 log.cc:1196] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/6e93e121a8854902b54f06c983ba5b9c
I20240501 14:09:46.653173 5192 ts_tablet_manager.cc:1909] T 6e93e121a8854902b54f06c983ba5b9c P db935501e9104de5b15c8780681d6c38: Deleting consensus metadata
I20240501 14:09:46.654951 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 6e93e121a8854902b54f06c983ba5b9c (table rebalancer_dynamic_table_new_name [id=6fab5ac330974a39be5a7c7e5774950a]) successfully deleted
I20240501 14:09:46.655128 5350 tablet_replica.cc:330] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:46.655151 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet 6e93e121a8854902b54f06c983ba5b9c (table rebalancer_dynamic_table_new_name [id=6fab5ac330974a39be5a7c7e5774950a]) successfully deleted
I20240501 14:09:46.655632 5350 raft_consensus.cc:2229] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:46.656186 5350 raft_consensus.cc:2258] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:46.656761 5350 ts_tablet_manager.cc:1875] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:46.659509 5350 ts_tablet_manager.cc:1888] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:46.659629 5350 log.cc:1196] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/6e93e121a8854902b54f06c983ba5b9c
I20240501 14:09:46.660699 5350 ts_tablet_manager.cc:1909] T 6e93e121a8854902b54f06c983ba5b9c P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:46.661339 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet 6e93e121a8854902b54f06c983ba5b9c (table rebalancer_dynamic_table_new_name [id=6fab5ac330974a39be5a7c7e5774950a]) successfully deleted
I20240501 14:09:46.681317 30388 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:46.683907 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:46.690225 29995 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:46.690457 4829 consensus_queue.cc:1035] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.690997 30124 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:46.694684 32664 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:46.704293 329 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:46.709669 32400 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:46.718983 32533 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:46.726226 4830 consensus_queue.cc:1035] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.754868 29873 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:46.759918 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 5d20daa5b420407897a0b9f468c7ab16 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e13c68e7e95a418db8a4d906a8e311b2]), partition=
I20240501 14:09:46.760037 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 5d20daa5b420407897a0b9f468c7ab16 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e13c68e7e95a418db8a4d906a8e311b2]), partition=
I20240501 14:09:46.760205 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d20daa5b420407897a0b9f468c7ab16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.760272 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d20daa5b420407897a0b9f468c7ab16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.762450 5374 tablet_bootstrap.cc:492] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:46.762671 5375 tablet_bootstrap.cc:492] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:46.762986 30124 tablet_service.cc:1467] Processing CreateTablet for tablet 5d20daa5b420407897a0b9f468c7ab16 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e13c68e7e95a418db8a4d906a8e311b2]), partition=
I20240501 14:09:46.763262 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d20daa5b420407897a0b9f468c7ab16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.765302 5376 tablet_bootstrap.cc:492] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:46.766311 5376 tablet_bootstrap.cc:654] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.767498 5375 tablet_bootstrap.cc:654] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.768421 5374 tablet_bootstrap.cc:654] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.769112 5376 tablet_bootstrap.cc:492] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:46.769604 5375 tablet_bootstrap.cc:492] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:46.769690 5375 ts_tablet_manager.cc:1367] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:46.769248 5376 ts_tablet_manager.cc:1367] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:46.770105 5375 raft_consensus.cc:348] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:46.770234 5375 raft_consensus.cc:374] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.770275 5375 raft_consensus.cc:729] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.770382 5375 consensus_queue.cc:260] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:46.770684 5375 ts_tablet_manager.cc:1398] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:46.770668 5376 raft_consensus.cc:348] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:46.770860 5376 raft_consensus.cc:374] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.770944 5376 raft_consensus.cc:729] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.771122 5376 consensus_queue.cc:260] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:46.771361 5374 tablet_bootstrap.cc:492] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:46.771498 5374 ts_tablet_manager.cc:1367] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:46.772156 5374 raft_consensus.cc:348] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:46.772362 5376 ts_tablet_manager.cc:1398] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:46.772365 5374 raft_consensus.cc:374] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.772413 5374 raft_consensus.cc:729] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.772563 5374 consensus_queue.cc:260] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:46.772851 5374 ts_tablet_manager.cc:1398] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:46.777262 30277 consensus_queue.cc:237] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:46.778486 349 raft_consensus.cc:1261] T 7e1895c0656d49c09266cdbd231497cb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:46.778578 30013 raft_consensus.cc:1261] T 7e1895c0656d49c09266cdbd231497cb P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:46.778862 5386 consensus_queue.cc:1035] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:46.779054 5117 consensus_queue.cc:1035] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:46.780894 5389 raft_consensus.cc:2932] T 7e1895c0656d49c09266cdbd231497cb P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:46.781337 30013 raft_consensus.cc:2932] T 7e1895c0656d49c09266cdbd231497cb P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:46.781337 349 raft_consensus.cc:2932] T 7e1895c0656d49c09266cdbd231497cb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:46.784862 29873 catalog_manager.cc:5484] T 7e1895c0656d49c09266cdbd231497cb P 93f2518fbadf4f9991cda93afacb363e reported cstate change: config changed from index 4 to 5, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:46.787568 30388 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1895c0656d49c09266cdbd231497cb with delete_type TABLET_DATA_TOMBSTONED (TS d1c217469ae4478ebd49e4ed9f7352d7 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:46.787784 5230 raft_consensus.cc:482] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:46.787881 5230 raft_consensus.cc:504] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:46.788317 5230 leader_election.cc:288] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:46.788668 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d20daa5b420407897a0b9f468c7ab16" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:46.788797 30146 raft_consensus.cc:2454] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:46.788782 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d20daa5b420407897a0b9f468c7ab16" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:46.788898 32684 raft_consensus.cc:2454] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:46.789177 32359 leader_election.cc:302] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:46.789347 5230 raft_consensus.cc:2781] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:46.789407 5230 raft_consensus.cc:482] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:46.789443 5230 raft_consensus.cc:3037] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.790448 5230 raft_consensus.cc:504] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:46.790786 5230 leader_election.cc:288] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:46.791316 5391 tablet_replica.cc:330] T 7e1895c0656d49c09266cdbd231497cb P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:46.791422 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d20daa5b420407897a0b9f468c7ab16" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:46.791502 5391 raft_consensus.cc:2229] T 7e1895c0656d49c09266cdbd231497cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:46.791517 30146 raft_consensus.cc:3037] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.791649 5391 raft_consensus.cc:2258] T 7e1895c0656d49c09266cdbd231497cb P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:46.792129 5391 ts_tablet_manager.cc:1875] T 7e1895c0656d49c09266cdbd231497cb P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:46.792426 30146 raft_consensus.cc:2454] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:46.792968 32359 leader_election.cc:302] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:46.793135 5230 raft_consensus.cc:2781] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:46.793213 5230 raft_consensus.cc:686] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:46.793346 5230 consensus_queue.cc:237] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:46.795320 29866 catalog_manager.cc:5484] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:46.796631 5391 ts_tablet_manager.cc:1888] T 7e1895c0656d49c09266cdbd231497cb P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:46.796698 5391 log.cc:1196] T 7e1895c0656d49c09266cdbd231497cb P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/7e1895c0656d49c09266cdbd231497cb
I20240501 14:09:46.797230 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 7e1895c0656d49c09266cdbd231497cb (table rebalance_test_table_11 [id=0ceb652d71d942b2a6e0ef7686e7635b]) successfully deleted
I20240501 14:09:46.800454 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d20daa5b420407897a0b9f468c7ab16" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:46.800572 32684 raft_consensus.cc:3037] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.801437 32684 raft_consensus.cc:2454] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:46.821842 29866 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:46.828161 32533 tablet_service.cc:1467] Processing CreateTablet for tablet 28500f605e0644b69207077f57023ca6 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=f4bc8404fe4a4d7a89d0369621ce5f0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.828207 32532 tablet_service.cc:1467] Processing CreateTablet for tablet 440f03e9113e46f6ba4f019071ebdacb (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=f4bc8404fe4a4d7a89d0369621ce5f0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.828505 32532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 440f03e9113e46f6ba4f019071ebdacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.829013 32402 tablet_service.cc:1467] Processing CreateTablet for tablet 440f03e9113e46f6ba4f019071ebdacb (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=f4bc8404fe4a4d7a89d0369621ce5f0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.829010 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 28500f605e0644b69207077f57023ca6 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=f4bc8404fe4a4d7a89d0369621ce5f0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.829128 329 tablet_service.cc:1467] Processing CreateTablet for tablet 440f03e9113e46f6ba4f019071ebdacb (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=f4bc8404fe4a4d7a89d0369621ce5f0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.829344 32402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 440f03e9113e46f6ba4f019071ebdacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.829375 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 440f03e9113e46f6ba4f019071ebdacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.829505 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28500f605e0644b69207077f57023ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.831436 5374 tablet_bootstrap.cc:492] T 440f03e9113e46f6ba4f019071ebdacb P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:46.831656 32401 tablet_service.cc:1467] Processing CreateTablet for tablet 5025e07cbfd543e3b4722c6c69e4a6b3 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=f4bc8404fe4a4d7a89d0369621ce5f0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.831890 32401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5025e07cbfd543e3b4722c6c69e4a6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.832419 5374 tablet_bootstrap.cc:654] T 440f03e9113e46f6ba4f019071ebdacb P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.832959 5414 tablet_bootstrap.cc:492] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:46.833379 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28500f605e0644b69207077f57023ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.833855 5414 tablet_bootstrap.cc:654] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.835343 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 28500f605e0644b69207077f57023ca6 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=f4bc8404fe4a4d7a89d0369621ce5f0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.835592 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28500f605e0644b69207077f57023ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.835759 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 5025e07cbfd543e3b4722c6c69e4a6b3 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=f4bc8404fe4a4d7a89d0369621ce5f0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.836017 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5025e07cbfd543e3b4722c6c69e4a6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.837172 32533 tablet_service.cc:1467] Processing CreateTablet for tablet 5025e07cbfd543e3b4722c6c69e4a6b3 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=f4bc8404fe4a4d7a89d0369621ce5f0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.837404 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5025e07cbfd543e3b4722c6c69e4a6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.838083 5375 tablet_bootstrap.cc:492] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:46.838685 5418 tablet_bootstrap.cc:492] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:46.839367 5375 tablet_bootstrap.cc:654] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.839596 5418 tablet_bootstrap.cc:654] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.840708 5418 tablet_bootstrap.cc:492] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:46.840809 5418 ts_tablet_manager.cc:1367] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:46.841275 5418 raft_consensus.cc:348] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:46.841382 5418 raft_consensus.cc:374] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.841408 5418 raft_consensus.cc:729] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.841511 5418 consensus_queue.cc:260] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:46.841836 5418 ts_tablet_manager.cc:1398] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:46.842671 5375 tablet_bootstrap.cc:492] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:46.842763 5375 ts_tablet_manager.cc:1367] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:46.843250 5375 raft_consensus.cc:348] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:46.843366 5375 raft_consensus.cc:374] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.843410 5375 raft_consensus.cc:729] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.843541 5375 consensus_queue.cc:260] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:46.843811 5375 ts_tablet_manager.cc:1398] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:46.846851 5413 tablet_bootstrap.cc:492] T 440f03e9113e46f6ba4f019071ebdacb P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:46.847105 5374 tablet_bootstrap.cc:492] T 440f03e9113e46f6ba4f019071ebdacb P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:46.847191 5374 ts_tablet_manager.cc:1367] T 440f03e9113e46f6ba4f019071ebdacb P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:09:46.847620 5374 raft_consensus.cc:348] T 440f03e9113e46f6ba4f019071ebdacb P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:46.847752 5413 tablet_bootstrap.cc:654] T 440f03e9113e46f6ba4f019071ebdacb P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.847769 5374 raft_consensus.cc:374] T 440f03e9113e46f6ba4f019071ebdacb P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.847810 5374 raft_consensus.cc:729] T 440f03e9113e46f6ba4f019071ebdacb P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.847940 5374 consensus_queue.cc:260] T 440f03e9113e46f6ba4f019071ebdacb P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:46.848279 5374 ts_tablet_manager.cc:1398] T 440f03e9113e46f6ba4f019071ebdacb P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:46.848469 5374 tablet_bootstrap.cc:492] T 28500f605e0644b69207077f57023ca6 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:46.848929 5413 tablet_bootstrap.cc:492] T 440f03e9113e46f6ba4f019071ebdacb P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:46.848995 5413 ts_tablet_manager.cc:1367] T 440f03e9113e46f6ba4f019071ebdacb P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:46.849043 5414 tablet_bootstrap.cc:492] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:46.849123 5414 ts_tablet_manager.cc:1367] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:09:46.849362 5413 raft_consensus.cc:348] T 440f03e9113e46f6ba4f019071ebdacb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:46.849450 5413 raft_consensus.cc:374] T 440f03e9113e46f6ba4f019071ebdacb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.849476 5413 raft_consensus.cc:729] T 440f03e9113e46f6ba4f019071ebdacb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.849584 5413 consensus_queue.cc:260] T 440f03e9113e46f6ba4f019071ebdacb P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:46.849639 5414 raft_consensus.cc:348] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:46.849735 5374 tablet_bootstrap.cc:654] T 28500f605e0644b69207077f57023ca6 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.849750 5414 raft_consensus.cc:374] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.849779 5414 raft_consensus.cc:729] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.849920 5413 ts_tablet_manager.cc:1398] T 440f03e9113e46f6ba4f019071ebdacb P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:46.849893 5414 consensus_queue.cc:260] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:46.850090 5413 tablet_bootstrap.cc:492] T 28500f605e0644b69207077f57023ca6 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:46.850215 5414 ts_tablet_manager.cc:1398] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:46.851645 5374 tablet_bootstrap.cc:492] T 28500f605e0644b69207077f57023ca6 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:46.851724 5374 ts_tablet_manager.cc:1367] T 28500f605e0644b69207077f57023ca6 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:46.851895 5413 tablet_bootstrap.cc:654] T 28500f605e0644b69207077f57023ca6 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.852150 5374 raft_consensus.cc:348] T 28500f605e0644b69207077f57023ca6 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:46.852272 5374 raft_consensus.cc:374] T 28500f605e0644b69207077f57023ca6 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.852319 5374 raft_consensus.cc:729] T 28500f605e0644b69207077f57023ca6 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.852435 5374 consensus_queue.cc:260] T 28500f605e0644b69207077f57023ca6 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:46.852738 5374 ts_tablet_manager.cc:1398] T 28500f605e0644b69207077f57023ca6 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:46.852902 5374 tablet_bootstrap.cc:492] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:46.853739 5374 tablet_bootstrap.cc:654] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.855597 5374 tablet_bootstrap.cc:492] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:46.855717 5374 ts_tablet_manager.cc:1367] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:46.856158 5374 raft_consensus.cc:348] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:46.856308 5374 raft_consensus.cc:374] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.856360 5374 raft_consensus.cc:729] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.856513 5374 consensus_queue.cc:260] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:46.858598 5374 ts_tablet_manager.cc:1398] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:46.861176 5413 tablet_bootstrap.cc:492] T 28500f605e0644b69207077f57023ca6 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:46.861301 5413 ts_tablet_manager.cc:1367] T 28500f605e0644b69207077f57023ca6 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:09:46.861728 5413 raft_consensus.cc:348] T 28500f605e0644b69207077f57023ca6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:46.861902 5413 raft_consensus.cc:374] T 28500f605e0644b69207077f57023ca6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.861979 5413 raft_consensus.cc:729] T 28500f605e0644b69207077f57023ca6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.862141 5413 consensus_queue.cc:260] T 28500f605e0644b69207077f57023ca6 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:46.862483 5413 ts_tablet_manager.cc:1398] T 28500f605e0644b69207077f57023ca6 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:46.862699 5413 tablet_bootstrap.cc:492] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:46.863821 5413 tablet_bootstrap.cc:654] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.864979 5413 tablet_bootstrap.cc:492] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:46.865069 5413 ts_tablet_manager.cc:1367] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:46.865437 5413 raft_consensus.cc:348] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:46.865514 5413 raft_consensus.cc:374] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.865538 5413 raft_consensus.cc:729] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.865649 5413 consensus_queue.cc:260] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:46.865864 5413 ts_tablet_manager.cc:1398] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:46.900204 29872 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:46.904544 30146 raft_consensus.cc:1261] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:46.905097 4825 consensus_queue.cc:1035] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.905546 32683 raft_consensus.cc:1261] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:46.907023 4825 consensus_queue.cc:1035] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.909792 5072 tablet.cc:1722] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:46.910127 5072 tablet.cc:1620] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:46.910130 29872 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet 5d20daa5b420407897a0b9f468c7ab16 (table rebalancer_dynamic_table [id=e13c68e7e95a418db8a4d906a8e311b2]). Expected version 1 got 0
I20240501 14:09:46.911486 5071 tablet.cc:1722] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:46.911788 5071 tablet.cc:1620] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:46.911929 29870 catalog_manager.cc:5576] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457) does not have the latest schema for tablet 5d20daa5b420407897a0b9f468c7ab16 (table rebalancer_dynamic_table [id=e13c68e7e95a418db8a4d906a8e311b2]). Expected version 1 got 0
I20240501 14:09:46.914357 4830 raft_consensus.cc:482] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:46.914443 4830 raft_consensus.cc:504] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:46.914839 4830 leader_election.cc:288] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:46.915539 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025e07cbfd543e3b4722c6c69e4a6b3" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:46.915678 32554 raft_consensus.cc:2454] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:46.916077 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025e07cbfd543e3b4722c6c69e4a6b3" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:46.916247 32416 raft_consensus.cc:2454] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:46.917730 32621 leader_election.cc:302] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:46.918681 29867 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=e13c68e7e95a418db8a4d906a8e311b2] alter complete (version 1)
I20240501 14:09:46.919006 4830 raft_consensus.cc:2781] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:46.919082 4830 raft_consensus.cc:482] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:46.919123 4830 raft_consensus.cc:3037] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.920248 4830 raft_consensus.cc:504] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:46.920609 4830 leader_election.cc:288] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:46.921085 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025e07cbfd543e3b4722c6c69e4a6b3" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:46.921208 32554 raft_consensus.cc:3037] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.921188 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025e07cbfd543e3b4722c6c69e4a6b3" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:46.921285 32416 raft_consensus.cc:3037] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.922258 32554 raft_consensus.cc:2454] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:46.922272 32416 raft_consensus.cc:2454] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:46.922636 32620 leader_election.cc:302] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:46.923431 4830 raft_consensus.cc:2781] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:46.923512 4830 raft_consensus.cc:686] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Becoming Leader. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
I20240501 14:09:46.923642 4830 consensus_queue.cc:237] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:46.926484 29867 catalog_manager.cc:5484] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db935501e9104de5b15c8780681d6c38 (127.22.252.135). New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:46.928829 29872 catalog_manager.cc:5576] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457) does not have the latest schema for tablet 5d20daa5b420407897a0b9f468c7ab16 (table rebalancer_dynamic_table [id=e13c68e7e95a418db8a4d906a8e311b2]). Expected version 1 got 0
I20240501 14:09:46.930449 5460 tablet.cc:1722] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:46.930752 5460 tablet.cc:1620] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3: MemRowSet was empty: no flush needed.
I20240501 14:09:46.990830 4756 raft_consensus.cc:482] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:46.990931 4756 raft_consensus.cc:504] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:46.991304 4756 leader_election.cc:288] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:46.991650 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "440f03e9113e46f6ba4f019071ebdacb" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:46.991650 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "440f03e9113e46f6ba4f019071ebdacb" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:46.991777 32554 raft_consensus.cc:2454] T 440f03e9113e46f6ba4f019071ebdacb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:46.991777 32416 raft_consensus.cc:2454] T 440f03e9113e46f6ba4f019071ebdacb P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:46.992084 32753 leader_election.cc:302] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:46.992344 4756 raft_consensus.cc:2781] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:46.992394 4756 raft_consensus.cc:482] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:46.992444 4756 raft_consensus.cc:3037] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.993252 4756 raft_consensus.cc:504] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:46.993600 4756 leader_election.cc:288] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:46.994000 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "440f03e9113e46f6ba4f019071ebdacb" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:46.994001 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "440f03e9113e46f6ba4f019071ebdacb" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:46.994098 32554 raft_consensus.cc:3037] T 440f03e9113e46f6ba4f019071ebdacb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.994098 32416 raft_consensus.cc:3037] T 440f03e9113e46f6ba4f019071ebdacb P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.995026 32554 raft_consensus.cc:2454] T 440f03e9113e46f6ba4f019071ebdacb P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:46.995026 32416 raft_consensus.cc:2454] T 440f03e9113e46f6ba4f019071ebdacb P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:46.995301 32753 leader_election.cc:302] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:46.995548 4756 raft_consensus.cc:2781] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:46.995700 4756 raft_consensus.cc:686] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:46.995827 4756 consensus_queue.cc:237] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:46.997757 29869 catalog_manager.cc:5484] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:47.020800 29869 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:47.025694 5072 tablet.cc:1722] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:47.026024 5072 tablet.cc:1620] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:47.026263 5071 tablet.cc:1722] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:47.026541 5071 tablet.cc:1620] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:47.027864 5460 tablet.cc:1722] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:47.028461 5460 tablet.cc:1620] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3: MemRowSet was empty: no flush needed.
I20240501 14:09:47.031491 29870 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=e13c68e7e95a418db8a4d906a8e311b2] alter complete (version 2)
I20240501 14:09:47.133406 29870 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:47.138269 5072 tablet.cc:1722] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:47.138828 5071 tablet.cc:1722] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:47.139973 5460 tablet.cc:1722] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:47.144084 29869 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=e13c68e7e95a418db8a4d906a8e311b2] alter complete (version 3)
I20240501 14:09:47.154433 4740 raft_consensus.cc:482] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:47.154531 4740 raft_consensus.cc:504] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:47.154871 4740 leader_election.cc:288] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:47.155193 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28500f605e0644b69207077f57023ca6" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:47.155230 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28500f605e0644b69207077f57023ca6" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:47.155323 32554 raft_consensus.cc:2454] T 28500f605e0644b69207077f57023ca6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:47.155342 32416 raft_consensus.cc:2454] T 28500f605e0644b69207077f57023ca6 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:47.155573 30344 leader_election.cc:302] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:47.155839 5464 raft_consensus.cc:2781] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:47.155895 5464 raft_consensus.cc:482] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:47.155953 5464 raft_consensus.cc:3037] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.156852 5464 raft_consensus.cc:504] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:47.157157 5464 leader_election.cc:288] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:47.157400 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28500f605e0644b69207077f57023ca6" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:47.157495 32554 raft_consensus.cc:3037] T 28500f605e0644b69207077f57023ca6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.157465 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28500f605e0644b69207077f57023ca6" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:47.157577 32416 raft_consensus.cc:3037] T 28500f605e0644b69207077f57023ca6 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.158304 32554 raft_consensus.cc:2454] T 28500f605e0644b69207077f57023ca6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:47.158483 32416 raft_consensus.cc:2454] T 28500f605e0644b69207077f57023ca6 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:47.158592 30344 leader_election.cc:302] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:47.158756 5464 raft_consensus.cc:2781] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:47.158861 5464 raft_consensus.cc:686] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:47.159070 5464 consensus_queue.cc:237] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:47.161115 29869 catalog_manager.cc:5484] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:47.182147 32533 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:47.184315 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:47.187616 29995 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:47.188082 32401 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:47.207373 32664 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:47.214200 329 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:47.233924 30124 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:47.243278 30388 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:47.247148 29869 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:47.247296 29869 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:47.253083 29869 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 5d20daa5b420407897a0b9f468c7ab16
I20240501 14:09:47.253633 32664 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d20daa5b420407897a0b9f468c7ab16 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:47 UTC) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:47.253886 30124 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d20daa5b420407897a0b9f468c7ab16 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:47 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:47.253917 32401 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d20daa5b420407897a0b9f468c7ab16 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:47 UTC) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:47.254202 5478 tablet_replica.cc:330] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:47.254601 5478 raft_consensus.cc:2229] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:47.254879 5478 raft_consensus.cc:2258] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:47.255893 5479 tablet_replica.cc:330] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:47.256111 5479 raft_consensus.cc:2229] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:47.256321 5479 raft_consensus.cc:2258] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:47.255587 5478 ts_tablet_manager.cc:1875] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:47.256947 5479 ts_tablet_manager.cc:1875] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:47.257153 5477 tablet_replica.cc:330] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:47.257375 5477 raft_consensus.cc:2229] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:47.257548 5477 raft_consensus.cc:2258] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:47.258261 5477 ts_tablet_manager.cc:1875] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:47.259358 5479 ts_tablet_manager.cc:1888] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:47.259479 5479 log.cc:1196] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/5d20daa5b420407897a0b9f468c7ab16
I20240501 14:09:47.259892 5479 ts_tablet_manager.cc:1909] T 5d20daa5b420407897a0b9f468c7ab16 P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:47.259915 5478 ts_tablet_manager.cc:1888] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:47.260028 5478 log.cc:1196] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/5d20daa5b420407897a0b9f468c7ab16
I20240501 14:09:47.260368 5478 ts_tablet_manager.cc:1909] T 5d20daa5b420407897a0b9f468c7ab16 P a91029b90bf9415a89a2f70736759ad3: Deleting consensus metadata
I20240501 14:09:47.260774 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 5d20daa5b420407897a0b9f468c7ab16 (table rebalancer_dynamic_table_new_name [id=e13c68e7e95a418db8a4d906a8e311b2]) successfully deleted
I20240501 14:09:47.261066 5477 ts_tablet_manager.cc:1888] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:47.261083 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 5d20daa5b420407897a0b9f468c7ab16 (table rebalancer_dynamic_table_new_name [id=e13c68e7e95a418db8a4d906a8e311b2]) successfully deleted
I20240501 14:09:47.261143 5477 log.cc:1196] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/5d20daa5b420407897a0b9f468c7ab16
I20240501 14:09:47.261385 5477 ts_tablet_manager.cc:1909] T 5d20daa5b420407897a0b9f468c7ab16 P db935501e9104de5b15c8780681d6c38: Deleting consensus metadata
I20240501 14:09:47.261950 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet 5d20daa5b420407897a0b9f468c7ab16 (table rebalancer_dynamic_table_new_name [id=e13c68e7e95a418db8a4d906a8e311b2]) successfully deleted
I20240501 14:09:47.283273 30406 consensus_queue.cc:237] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:47.284462 32683 raft_consensus.cc:1261] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:47.284508 30277 raft_consensus.cc:1261] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:47.284991 4740 consensus_queue.cc:1035] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:47.285405 4740 consensus_queue.cc:1035] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:47.286728 5464 raft_consensus.cc:2932] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:47.287161 32683 raft_consensus.cc:2932] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:47.287663 30277 raft_consensus.cc:2932] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:47.291035 29869 catalog_manager.cc:5484] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38 reported cstate change: config changed from index 4 to 6, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New cstate: current_term: 2 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:47.296597 30124 tablet_service.cc:1514] Processing DeleteTablet for tablet c362bd01a59b4bcf85eeb59713e58b64 with delete_type TABLET_DATA_TOMBSTONED (TS d35c16dcb506426993811bb2519d7db3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:47.297034 5479 tablet_replica.cc:330] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:47.297200 5479 raft_consensus.cc:2229] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:47.297310 5479 raft_consensus.cc:2258] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:47.297902 5479 ts_tablet_manager.cc:1875] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:47.300011 5479 ts_tablet_manager.cc:1888] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 14:09:47.300091 5479 log.cc:1196] T c362bd01a59b4bcf85eeb59713e58b64 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/c362bd01a59b4bcf85eeb59713e58b64
I20240501 14:09:47.301604 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet c362bd01a59b4bcf85eeb59713e58b64 (table rebalance_test_table_13 [id=422856ed458a4c52950e8e3e49019a55]) successfully deleted
I20240501 14:09:47.355113 29872 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:47.360693 32533 tablet_service.cc:1467] Processing CreateTablet for tablet 2fa79f44ebe34c4b8b1b1ffdb9685572 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55a5f4150d214a30b3731786a3fa1daf]), partition=
I20240501 14:09:47.360975 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fa79f44ebe34c4b8b1b1ffdb9685572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.361253 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 2fa79f44ebe34c4b8b1b1ffdb9685572 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55a5f4150d214a30b3731786a3fa1daf]), partition=
I20240501 14:09:47.361500 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fa79f44ebe34c4b8b1b1ffdb9685572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.363240 5413 tablet_bootstrap.cc:492] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:47.363826 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 2fa79f44ebe34c4b8b1b1ffdb9685572 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55a5f4150d214a30b3731786a3fa1daf]), partition=
I20240501 14:09:47.364058 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fa79f44ebe34c4b8b1b1ffdb9685572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.364173 5413 tablet_bootstrap.cc:654] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.365409 5413 tablet_bootstrap.cc:492] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:47.365494 5413 ts_tablet_manager.cc:1367] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:47.365901 5413 raft_consensus.cc:348] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:47.366011 5413 raft_consensus.cc:374] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.366047 5413 raft_consensus.cc:729] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.366183 5413 consensus_queue.cc:260] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:47.366346 5548 tablet_bootstrap.cc:492] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:47.366451 5413 ts_tablet_manager.cc:1398] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:47.368736 5546 tablet_bootstrap.cc:492] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:47.369813 5546 tablet_bootstrap.cc:654] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.373183 5546 tablet_bootstrap.cc:492] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:47.373332 5546 ts_tablet_manager.cc:1367] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:47.373793 5546 raft_consensus.cc:348] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:47.373900 5546 raft_consensus.cc:374] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.373945 5546 raft_consensus.cc:729] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.374078 5546 consensus_queue.cc:260] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:47.374361 5546 ts_tablet_manager.cc:1398] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:47.378453 4830 consensus_queue.cc:1035] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:47.380321 5548 tablet_bootstrap.cc:654] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.383396 5070 consensus_queue.cc:1035] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:47.384387 5548 tablet_bootstrap.cc:492] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:47.384496 5548 ts_tablet_manager.cc:1367] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.001s
I20240501 14:09:47.384944 5548 raft_consensus.cc:348] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:47.385077 5548 raft_consensus.cc:374] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.385128 5548 raft_consensus.cc:729] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.385488 5548 consensus_queue.cc:260] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:47.386072 5548 ts_tablet_manager.cc:1398] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:47.416637 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:47.417133 32533 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:47.419656 29995 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:47.422271 329 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:47.423112 32664 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:47.431712 32401 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:47.435274 5464 raft_consensus.cc:482] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:47.435362 5464 raft_consensus.cc:504] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:47.435703 5464 leader_election.cc:288] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:47.435985 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fa79f44ebe34c4b8b1b1ffdb9685572" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:47.436040 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fa79f44ebe34c4b8b1b1ffdb9685572" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:47.436121 32683 raft_consensus.cc:2454] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:47.436141 32554 raft_consensus.cc:2454] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:47.436416 30344 leader_election.cc:302] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:47.436590 5464 raft_consensus.cc:2781] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:47.436640 5464 raft_consensus.cc:482] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:47.436678 5464 raft_consensus.cc:3037] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.437819 5464 raft_consensus.cc:504] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:47.438105 5464 leader_election.cc:288] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:47.438542 4756 consensus_queue.cc:1035] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:47.442644 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fa79f44ebe34c4b8b1b1ffdb9685572" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:47.442754 32553 raft_consensus.cc:3037] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.443686 32553 raft_consensus.cc:2454] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:47.444011 30344 leader_election.cc:302] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:47.444391 4740 raft_consensus.cc:2781] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:47.444461 4740 raft_consensus.cc:686] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:47.444571 4740 consensus_queue.cc:237] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:47.447042 29872 catalog_manager.cc:5484] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:47.447651 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fa79f44ebe34c4b8b1b1ffdb9685572" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:47.447752 32683 raft_consensus.cc:3037] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.448805 32683 raft_consensus.cc:2454] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:47.460857 4756 consensus_queue.cc:1035] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:47.471822 30388 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:47.480715 30124 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:47.516984 30146 consensus_queue.cc:237] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:47.518182 32554 raft_consensus.cc:1261] T c7df6b8d76714eebbeacc85f403d3405 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:47.518352 32416 raft_consensus.cc:1261] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:47.518631 5567 consensus_queue.cc:1035] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:47.518790 5566 consensus_queue.cc:1035] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:47.520473 5567 raft_consensus.cc:2932] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:47.520830 32554 raft_consensus.cc:2932] T c7df6b8d76714eebbeacc85f403d3405 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:47.520993 32416 raft_consensus.cc:2932] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:47.523931 29872 catalog_manager.cc:5484] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:47.526824 32664 tablet_service.cc:1514] Processing DeleteTablet for tablet c7df6b8d76714eebbeacc85f403d3405 with delete_type TABLET_DATA_TOMBSTONED (TS db935501e9104de5b15c8780681d6c38 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:47.527098 5477 tablet_replica.cc:330] T c7df6b8d76714eebbeacc85f403d3405 P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:47.527259 5477 raft_consensus.cc:2229] T c7df6b8d76714eebbeacc85f403d3405 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:47.527385 5477 raft_consensus.cc:2258] T c7df6b8d76714eebbeacc85f403d3405 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:47.528115 5477 ts_tablet_manager.cc:1875] T c7df6b8d76714eebbeacc85f403d3405 P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:47.530246 5477 ts_tablet_manager.cc:1888] T c7df6b8d76714eebbeacc85f403d3405 P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:09:47.530318 5477 log.cc:1196] T c7df6b8d76714eebbeacc85f403d3405 P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/c7df6b8d76714eebbeacc85f403d3405
I20240501 14:09:47.530841 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet c7df6b8d76714eebbeacc85f403d3405 (table rebalancer_extra_table_8 [id=8ff8f244c547410f853c771c11ecdd1e]) successfully deleted
I20240501 14:09:47.553658 29865 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:47.562019 32683 raft_consensus.cc:1261] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:47.562168 32554 raft_consensus.cc:1261] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:47.562463 5464 consensus_queue.cc:1035] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:47.562685 4740 consensus_queue.cc:1035] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:47.565078 29865 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet 2fa79f44ebe34c4b8b1b1ffdb9685572 (table rebalancer_dynamic_table [id=55a5f4150d214a30b3731786a3fa1daf]). Expected version 1 got 0
I20240501 14:09:47.565459 29868 catalog_manager.cc:5576] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497) does not have the latest schema for tablet 2fa79f44ebe34c4b8b1b1ffdb9685572 (table rebalancer_dynamic_table [id=55a5f4150d214a30b3731786a3fa1daf]). Expected version 1 got 0
I20240501 14:09:47.566532 5621 tablet.cc:1722] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:47.566568 5071 tablet.cc:1722] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:47.566856 5071 tablet.cc:1620] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:47.567238 5621 tablet.cc:1620] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7: MemRowSet was empty: no flush needed.
I20240501 14:09:47.569841 5554 mvcc.cc:205] Tried to move back new op lower bound from 7022889318650740736 to 7022889318174134272. Current Snapshot: MvccSnapshot[applied={T|T < 7022889318650740736}]
I20240501 14:09:47.571498 29868 catalog_manager.cc:5576] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497) does not have the latest schema for tablet 2fa79f44ebe34c4b8b1b1ffdb9685572 (table rebalancer_dynamic_table [id=55a5f4150d214a30b3731786a3fa1daf]). Expected version 1 got 0
I20240501 14:09:47.574095 5622 tablet.cc:1722] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:47.573802 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=55a5f4150d214a30b3731786a3fa1daf] alter complete (version 1)
I20240501 14:09:47.574440 5622 tablet.cc:1620] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:47.634690 5464 consensus_queue.cc:1035] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:47.644582 4740 consensus_queue.cc:1035] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:47.666374 29995 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:47.671056 32533 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:47.674546 329 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:47.675292 29870 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:47.676663 32401 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:47.680160 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:47.684927 32664 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:47.685132 5621 tablet.cc:1722] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:47.685424 5621 tablet.cc:1620] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7: MemRowSet was empty: no flush needed.
I20240501 14:09:47.686671 30388 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:47.688545 5071 tablet.cc:1722] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:47.688859 5071 tablet.cc:1620] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:47.691035 5622 tablet.cc:1722] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:47.691325 5622 tablet.cc:1620] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:47.704145 30124 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:47.705220 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=55a5f4150d214a30b3731786a3fa1daf] alter complete (version 2)
I20240501 14:09:47.716563 29870 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:47.726338 329 tablet_service.cc:1467] Processing CreateTablet for tablet dbf1233541ad47eda949653a1b486dee (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b8abb09b9b7a47a3a002b43c042f0525]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:47.726435 32533 tablet_service.cc:1467] Processing CreateTablet for tablet 36bdef51cf8a4256bedd1102d04c7aef (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b8abb09b9b7a47a3a002b43c042f0525]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:47.726656 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbf1233541ad47eda949653a1b486dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.726701 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36bdef51cf8a4256bedd1102d04c7aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.727277 30124 tablet_service.cc:1467] Processing CreateTablet for tablet dbf1233541ad47eda949653a1b486dee (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b8abb09b9b7a47a3a002b43c042f0525]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:47.727528 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbf1233541ad47eda949653a1b486dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.727733 32664 tablet_service.cc:1467] Processing CreateTablet for tablet dbf1233541ad47eda949653a1b486dee (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b8abb09b9b7a47a3a002b43c042f0525]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:47.728010 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbf1233541ad47eda949653a1b486dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.728773 29995 tablet_service.cc:1467] Processing CreateTablet for tablet 14267ff763f5409998f457e14e407ed0 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b8abb09b9b7a47a3a002b43c042f0525]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:47.729029 29995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14267ff763f5409998f457e14e407ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.730360 5548 tablet_bootstrap.cc:492] T dbf1233541ad47eda949653a1b486dee P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:47.731354 5548 tablet_bootstrap.cc:654] T dbf1233541ad47eda949653a1b486dee P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.732488 30256 tablet_service.cc:1467] Processing CreateTablet for tablet 36bdef51cf8a4256bedd1102d04c7aef (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b8abb09b9b7a47a3a002b43c042f0525]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:47.732583 32401 tablet_service.cc:1467] Processing CreateTablet for tablet 14267ff763f5409998f457e14e407ed0 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b8abb09b9b7a47a3a002b43c042f0525]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:47.732774 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36bdef51cf8a4256bedd1102d04c7aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.732853 32401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14267ff763f5409998f457e14e407ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.734107 5413 tablet_bootstrap.cc:492] T 36bdef51cf8a4256bedd1102d04c7aef P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:47.734639 5666 tablet_bootstrap.cc:492] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:47.735098 5413 tablet_bootstrap.cc:654] T 36bdef51cf8a4256bedd1102d04c7aef P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.735280 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 14267ff763f5409998f457e14e407ed0 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b8abb09b9b7a47a3a002b43c042f0525]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:47.735513 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14267ff763f5409998f457e14e407ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.736008 5666 tablet_bootstrap.cc:654] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.737558 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 36bdef51cf8a4256bedd1102d04c7aef (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b8abb09b9b7a47a3a002b43c042f0525]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:47.737771 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36bdef51cf8a4256bedd1102d04c7aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.742916 5667 tablet_bootstrap.cc:492] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:47.743736 5672 tablet_bootstrap.cc:492] T dbf1233541ad47eda949653a1b486dee P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:47.745122 5667 tablet_bootstrap.cc:654] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.745944 5672 tablet_bootstrap.cc:654] T dbf1233541ad47eda949653a1b486dee P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.747263 5667 tablet_bootstrap.cc:492] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:47.747349 5667 ts_tablet_manager.cc:1367] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:47.747757 5667 raft_consensus.cc:348] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:47.747877 5667 raft_consensus.cc:374] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.747916 5667 raft_consensus.cc:729] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.748024 5667 consensus_queue.cc:260] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:47.748301 5667 ts_tablet_manager.cc:1398] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:47.748342 5670 tablet_bootstrap.cc:492] T 14267ff763f5409998f457e14e407ed0 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:47.749346 5548 tablet_bootstrap.cc:492] T dbf1233541ad47eda949653a1b486dee P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:47.749446 5548 ts_tablet_manager.cc:1367] T dbf1233541ad47eda949653a1b486dee P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s
I20240501 14:09:47.749732 5666 tablet_bootstrap.cc:492] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:47.749799 5666 ts_tablet_manager.cc:1367] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.001s
I20240501 14:09:47.749856 5548 raft_consensus.cc:348] T dbf1233541ad47eda949653a1b486dee P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:47.749955 5548 raft_consensus.cc:374] T dbf1233541ad47eda949653a1b486dee P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.749984 5548 raft_consensus.cc:729] T dbf1233541ad47eda949653a1b486dee P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.750095 5548 consensus_queue.cc:260] T dbf1233541ad47eda949653a1b486dee P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:47.750195 5666 raft_consensus.cc:348] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:47.750301 5666 raft_consensus.cc:374] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.750334 5666 raft_consensus.cc:729] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.750348 5548 ts_tablet_manager.cc:1398] T dbf1233541ad47eda949653a1b486dee P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:47.750514 5548 tablet_bootstrap.cc:492] T 14267ff763f5409998f457e14e407ed0 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:47.750521 5666 consensus_queue.cc:260] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:47.751549 5548 tablet_bootstrap.cc:654] T 14267ff763f5409998f457e14e407ed0 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.753489 5548 tablet_bootstrap.cc:492] T 14267ff763f5409998f457e14e407ed0 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:47.753561 5548 ts_tablet_manager.cc:1367] T 14267ff763f5409998f457e14e407ed0 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:47.753973 5548 raft_consensus.cc:348] T 14267ff763f5409998f457e14e407ed0 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:47.754068 5548 raft_consensus.cc:374] T 14267ff763f5409998f457e14e407ed0 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.754096 5548 raft_consensus.cc:729] T 14267ff763f5409998f457e14e407ed0 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.754221 5548 consensus_queue.cc:260] T 14267ff763f5409998f457e14e407ed0 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:47.754495 5548 ts_tablet_manager.cc:1398] T 14267ff763f5409998f457e14e407ed0 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:47.754622 5548 tablet_bootstrap.cc:492] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:47.755434 5548 tablet_bootstrap.cc:654] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.756400 5413 tablet_bootstrap.cc:492] T 36bdef51cf8a4256bedd1102d04c7aef P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:47.756497 5413 ts_tablet_manager.cc:1367] T 36bdef51cf8a4256bedd1102d04c7aef P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.001s
I20240501 14:09:47.756505 5548 tablet_bootstrap.cc:492] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:47.756574 5548 ts_tablet_manager.cc:1367] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:47.756902 5413 raft_consensus.cc:348] T 36bdef51cf8a4256bedd1102d04c7aef P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:47.756997 5413 raft_consensus.cc:374] T 36bdef51cf8a4256bedd1102d04c7aef P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.757025 5413 raft_consensus.cc:729] T 36bdef51cf8a4256bedd1102d04c7aef P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.757135 5413 consensus_queue.cc:260] T 36bdef51cf8a4256bedd1102d04c7aef P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:47.757277 5548 raft_consensus.cc:348] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:47.757385 5548 raft_consensus.cc:374] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.757391 5413 ts_tablet_manager.cc:1398] T 36bdef51cf8a4256bedd1102d04c7aef P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:47.757418 5548 raft_consensus.cc:729] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.757526 5548 consensus_queue.cc:260] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:47.757795 5548 ts_tablet_manager.cc:1398] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:47.758759 5669 tablet_bootstrap.cc:492] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:47.759644 5669 tablet_bootstrap.cc:654] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.759992 5666 ts_tablet_manager.cc:1398] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.010s user 0.000s sys 0.001s
I20240501 14:09:47.760609 5670 tablet_bootstrap.cc:654] T 14267ff763f5409998f457e14e407ed0 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.763087 5672 tablet_bootstrap.cc:492] T dbf1233541ad47eda949653a1b486dee P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:47.763186 5672 ts_tablet_manager.cc:1367] T dbf1233541ad47eda949653a1b486dee P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 14:09:47.763422 5669 tablet_bootstrap.cc:492] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:47.763494 5669 ts_tablet_manager.cc:1367] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:47.763588 5130 raft_consensus.cc:482] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:47.763670 5130 raft_consensus.cc:504] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:47.763651 5672 raft_consensus.cc:348] T dbf1233541ad47eda949653a1b486dee P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:47.763756 5672 raft_consensus.cc:374] T dbf1233541ad47eda949653a1b486dee P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.763787 5672 raft_consensus.cc:729] T dbf1233541ad47eda949653a1b486dee P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.763897 5672 consensus_queue.cc:260] T dbf1233541ad47eda949653a1b486dee P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:47.764034 5130 leader_election.cc:288] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:47.764019 5669 raft_consensus.cc:348] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:47.764117 5669 raft_consensus.cc:374] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.764147 5669 raft_consensus.cc:729] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.764171 5672 ts_tablet_manager.cc:1398] T dbf1233541ad47eda949653a1b486dee P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:47.764288 5669 consensus_queue.cc:260] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:47.764554 5669 ts_tablet_manager.cc:1398] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:47.764927 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14267ff763f5409998f457e14e407ed0" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:47.764931 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14267ff763f5409998f457e14e407ed0" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:47.765053 32684 raft_consensus.cc:2454] T 14267ff763f5409998f457e14e407ed0 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:47.765357 29948 leader_election.cc:302] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, db935501e9104de5b15c8780681d6c38; no voters:
W20240501 14:09:47.765633 29950 leader_election.cc:341] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:47.766285 5130 raft_consensus.cc:2781] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:47.766340 5130 raft_consensus.cc:482] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:47.766373 5130 raft_consensus.cc:3037] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.767498 5130 raft_consensus.cc:504] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:47.767820 5130 leader_election.cc:288] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:47.768213 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14267ff763f5409998f457e14e407ed0" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
W20240501 14:09:47.769160 29950 leader_election.cc:341] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:47.769333 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14267ff763f5409998f457e14e407ed0" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:47.769428 32684 raft_consensus.cc:3037] T 14267ff763f5409998f457e14e407ed0 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.770332 32684 raft_consensus.cc:2454] T 14267ff763f5409998f457e14e407ed0 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:47.770571 29948 leader_election.cc:302] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, db935501e9104de5b15c8780681d6c38; no voters: a91029b90bf9415a89a2f70736759ad3
I20240501 14:09:47.770747 5130 raft_consensus.cc:2781] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:47.770830 5130 raft_consensus.cc:686] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:47.770931 5130 consensus_queue.cc:237] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:47.772821 29870 catalog_manager.cc:5484] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:47.776871 5670 tablet_bootstrap.cc:492] T 14267ff763f5409998f457e14e407ed0 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:47.776978 5670 ts_tablet_manager.cc:1367] T 14267ff763f5409998f457e14e407ed0 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240501 14:09:47.777482 5670 raft_consensus.cc:348] T 14267ff763f5409998f457e14e407ed0 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:47.777596 5670 raft_consensus.cc:374] T 14267ff763f5409998f457e14e407ed0 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.777657 5670 raft_consensus.cc:729] T 14267ff763f5409998f457e14e407ed0 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.777841 5670 consensus_queue.cc:260] T 14267ff763f5409998f457e14e407ed0 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:47.778106 5670 ts_tablet_manager.cc:1398] T 14267ff763f5409998f457e14e407ed0 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:47.806465 30406 consensus_queue.cc:237] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:47.807758 349 raft_consensus.cc:1261] T e122adbba7c749cd93c53fa07a8a8b84 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:47.808158 30013 raft_consensus.cc:1261] T e122adbba7c749cd93c53fa07a8a8b84 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:47.808595 4740 consensus_queue.cc:1035] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:47.809605 5618 consensus_queue.cc:1035] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:47.811239 4740 raft_consensus.cc:2932] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:47.813001 349 raft_consensus.cc:2932] T e122adbba7c749cd93c53fa07a8a8b84 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:47.815685 29870 catalog_manager.cc:5484] T e122adbba7c749cd93c53fa07a8a8b84 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:47.817062 29872 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:47.821792 30014 raft_consensus.cc:2932] T e122adbba7c749cd93c53fa07a8a8b84 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:47.824030 5621 tablet.cc:1722] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:47.826013 32533 tablet_service.cc:1514] Processing DeleteTablet for tablet e122adbba7c749cd93c53fa07a8a8b84 with delete_type TABLET_DATA_TOMBSTONED (TS 5f8a0cfdfdca4148a75686dc88059cf3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:47.827255 5706 tablet_replica.cc:330] T e122adbba7c749cd93c53fa07a8a8b84 P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:47.827505 5070 raft_consensus.cc:482] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:47.827618 5070 raft_consensus.cc:504] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:47.828088 5070 leader_election.cc:288] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:47.828467 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bdef51cf8a4256bedd1102d04c7aef" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:47.828599 30277 raft_consensus.cc:2454] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:47.828895 32620 leader_election.cc:302] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:47.829126 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bdef51cf8a4256bedd1102d04c7aef" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:47.829246 32553 raft_consensus.cc:2454] T 36bdef51cf8a4256bedd1102d04c7aef P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:47.829638 5070 raft_consensus.cc:2781] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:47.829690 5070 raft_consensus.cc:482] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:47.829735 5070 raft_consensus.cc:3037] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.830288 5622 tablet.cc:1722] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:47.830586 5070 raft_consensus.cc:504] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:47.831004 5070 leader_election.cc:288] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:47.827445 5706 raft_consensus.cc:2229] T e122adbba7c749cd93c53fa07a8a8b84 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:47.831287 5706 raft_consensus.cc:2258] T e122adbba7c749cd93c53fa07a8a8b84 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:47.831672 5706 ts_tablet_manager.cc:1875] T e122adbba7c749cd93c53fa07a8a8b84 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:47.832521 5071 tablet.cc:1722] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:47.832839 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bdef51cf8a4256bedd1102d04c7aef" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:47.832964 30277 raft_consensus.cc:3037] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.833906 30277 raft_consensus.cc:2454] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:47.835275 5706 ts_tablet_manager.cc:1888] T e122adbba7c749cd93c53fa07a8a8b84 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:09:47.835338 5706 log.cc:1196] T e122adbba7c749cd93c53fa07a8a8b84 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/e122adbba7c749cd93c53fa07a8a8b84
I20240501 14:09:47.835698 32620 leader_election.cc:302] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:47.835887 5070 raft_consensus.cc:2781] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:47.835978 5070 raft_consensus.cc:686] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Becoming Leader. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
I20240501 14:09:47.836093 5070 consensus_queue.cc:237] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:47.836171 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet e122adbba7c749cd93c53fa07a8a8b84 (table rebalancer_extra_table_0 [id=25cba3c30d2b44599355b7317baedb1c]) successfully deleted
I20240501 14:09:47.836198 4756 raft_consensus.cc:482] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:47.836293 4756 raft_consensus.cc:504] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:47.836642 4756 leader_election.cc:288] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:47.836872 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbf1233541ad47eda949653a1b486dee" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:47.836989 32684 raft_consensus.cc:2454] T dbf1233541ad47eda949653a1b486dee P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:47.837100 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbf1233541ad47eda949653a1b486dee" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:47.837240 30146 raft_consensus.cc:2454] T dbf1233541ad47eda949653a1b486dee P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:47.837471 32750 leader_election.cc:302] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:47.837571 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bdef51cf8a4256bedd1102d04c7aef" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:47.837667 32554 raft_consensus.cc:3037] T 36bdef51cf8a4256bedd1102d04c7aef P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.837692 4756 raft_consensus.cc:2781] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:47.837747 4756 raft_consensus.cc:482] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:47.837775 4756 raft_consensus.cc:3037] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.838456 32554 raft_consensus.cc:2454] T 36bdef51cf8a4256bedd1102d04c7aef P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:47.838620 4756 raft_consensus.cc:504] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:47.838989 4756 leader_election.cc:288] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:47.839294 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbf1233541ad47eda949653a1b486dee" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:47.839409 30146 raft_consensus.cc:3037] T dbf1233541ad47eda949653a1b486dee P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.839417 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbf1233541ad47eda949653a1b486dee" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:47.839504 32684 raft_consensus.cc:3037] T dbf1233541ad47eda949653a1b486dee P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.840108 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=55a5f4150d214a30b3731786a3fa1daf] alter complete (version 3)
I20240501 14:09:47.840378 32684 raft_consensus.cc:2454] T dbf1233541ad47eda949653a1b486dee P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:47.840479 30146 raft_consensus.cc:2454] T dbf1233541ad47eda949653a1b486dee P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:47.840682 32750 leader_election.cc:302] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:47.840917 4756 raft_consensus.cc:2781] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:47.841013 4756 raft_consensus.cc:686] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:47.841137 4756 consensus_queue.cc:237] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:47.843091 29873 catalog_manager.cc:5484] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db935501e9104de5b15c8780681d6c38 (127.22.252.135). New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:47.843202 29870 catalog_manager.cc:5484] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:47.869875 30014 consensus_queue.cc:237] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:47.871203 30406 raft_consensus.cc:1261] T deee2ab875274d4eaeab84e698241ca8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:47.871382 30277 raft_consensus.cc:1261] T deee2ab875274d4eaeab84e698241ca8 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:47.871657 5730 consensus_queue.cc:1035] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:47.872022 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: deee2ab875274d4eaeab84e698241ca8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:47.872045 5680 consensus_queue.cc:1035] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:47.873889 5680 raft_consensus.cc:2932] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:47.874372 30406 raft_consensus.cc:2932] T deee2ab875274d4eaeab84e698241ca8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:47.877127 30264 raft_consensus.cc:2932] T deee2ab875274d4eaeab84e698241ca8 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:47.879837 29870 catalog_manager.cc:5484] T deee2ab875274d4eaeab84e698241ca8 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:47.896061 30406 consensus_queue.cc:237] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:47.897394 30264 raft_consensus.cc:1261] T 371c725533d84656b7a76c1bc981d038 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:47.897389 30146 raft_consensus.cc:1261] T 371c725533d84656b7a76c1bc981d038 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:47.897524 30343 consensus_peers.cc:485] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): Couldn't send request to peer 9ecd907cf5634ac98f3f54f3dbec6849. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 371c725533d84656b7a76c1bc981d038. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:47.897786 5618 consensus_queue.cc:1035] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:47.897943 4740 consensus_queue.cc:1035] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:47.899616 4740 raft_consensus.cc:2932] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:47.900036 30146 raft_consensus.cc:2932] T 371c725533d84656b7a76c1bc981d038 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:47.901331 30264 raft_consensus.cc:2932] T 371c725533d84656b7a76c1bc981d038 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:47.902293 29872 catalog_manager.cc:5484] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index -1 to 4, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:47.923004 30014 consensus_queue.cc:237] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:47.924546 32553 raft_consensus.cc:1261] T 7b3a90c03de64d6891dda267167d1d7b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:47.924628 32415 raft_consensus.cc:1261] T 7b3a90c03de64d6891dda267167d1d7b P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:47.924896 5680 consensus_queue.cc:1035] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:09:47.925055 29950 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): Couldn't send request to peer 1d0bb69611af4876a2b3b33c4766e13a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b3a90c03de64d6891dda267167d1d7b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:47.925063 5130 consensus_queue.cc:1035] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:47.926726 5680 raft_consensus.cc:2932] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:47.927096 32553 raft_consensus.cc:2932] T 7b3a90c03de64d6891dda267167d1d7b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:47.928426 32415 raft_consensus.cc:2932] T 7b3a90c03de64d6891dda267167d1d7b P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:47.930111 29873 catalog_manager.cc:5484] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e reported cstate change: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:47.942692 29870 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:47.942818 29870 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:47.945114 29870 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 2fa79f44ebe34c4b8b1b1ffdb9685572
I20240501 14:09:47.945883 30388 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fa79f44ebe34c4b8b1b1ffdb9685572 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:47 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:47.946076 32664 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fa79f44ebe34c4b8b1b1ffdb9685572 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:47 UTC) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:47.946262 5477 tablet_replica.cc:330] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:47.946301 32533 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fa79f44ebe34c4b8b1b1ffdb9685572 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:47 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:47.946419 5477 raft_consensus.cc:2229] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:47.946489 5786 tablet_replica.cc:330] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:47.946544 5477 raft_consensus.cc:2258] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:47.946625 5786 raft_consensus.cc:2229] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:47.946724 5706 tablet_replica.cc:330] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:47.946815 5786 raft_consensus.cc:2258] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:47.946852 5706 raft_consensus.cc:2229] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:47.946940 5706 raft_consensus.cc:2258] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:47.947332 5786 ts_tablet_manager.cc:1875] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:47.947446 5706 ts_tablet_manager.cc:1875] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:47.947731 5477 ts_tablet_manager.cc:1875] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:47.949579 5477 ts_tablet_manager.cc:1888] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:47.949605 5706 ts_tablet_manager.cc:1888] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:47.949651 5477 log.cc:1196] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/2fa79f44ebe34c4b8b1b1ffdb9685572
I20240501 14:09:47.949687 5706 log.cc:1196] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/2fa79f44ebe34c4b8b1b1ffdb9685572
I20240501 14:09:47.949852 5786 ts_tablet_manager.cc:1888] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:47.949892 5477 ts_tablet_manager.cc:1909] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P db935501e9104de5b15c8780681d6c38: Deleting consensus metadata
I20240501 14:09:47.949926 5786 log.cc:1196] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/2fa79f44ebe34c4b8b1b1ffdb9685572
I20240501 14:09:47.949965 5706 ts_tablet_manager.cc:1909] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:47.950260 5786 ts_tablet_manager.cc:1909] T 2fa79f44ebe34c4b8b1b1ffdb9685572 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:47.950721 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet 2fa79f44ebe34c4b8b1b1ffdb9685572 (table rebalancer_dynamic_table_new_name [id=55a5f4150d214a30b3731786a3fa1daf]) successfully deleted
I20240501 14:09:47.950884 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 2fa79f44ebe34c4b8b1b1ffdb9685572 (table rebalancer_dynamic_table_new_name [id=55a5f4150d214a30b3731786a3fa1daf]) successfully deleted
I20240501 14:09:47.951002 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet 2fa79f44ebe34c4b8b1b1ffdb9685572 (table rebalancer_dynamic_table_new_name [id=55a5f4150d214a30b3731786a3fa1daf]) successfully deleted
I20240501 14:09:47.952953 349 consensus_queue.cc:237] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
W20240501 14:09:47.954411 32753 consensus_peers.cc:485] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Couldn't send request to peer d35c16dcb506426993811bb2519d7db3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b66c71cbccfa4c96bdf2b438938bf56b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:47.954430 30014 raft_consensus.cc:1261] T b66c71cbccfa4c96bdf2b438938bf56b P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:47.954522 32553 raft_consensus.cc:1261] T b66c71cbccfa4c96bdf2b438938bf56b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:47.954874 5789 consensus_queue.cc:1035] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:47.955091 5788 consensus_queue.cc:1035] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:47.956779 5789 raft_consensus.cc:2932] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:47.957180 30014 raft_consensus.cc:2932] T b66c71cbccfa4c96bdf2b438938bf56b P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:47.958536 32553 raft_consensus.cc:2932] T b66c71cbccfa4c96bdf2b438938bf56b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:47.960197 29870 catalog_manager.cc:5484] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:47.980057 32415 consensus_queue.cc:237] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:47.981379 32684 raft_consensus.cc:1261] T b26b6d19699047afaee57d6f2c887664 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:47.981537 32553 raft_consensus.cc:1261] T b26b6d19699047afaee57d6f2c887664 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:47.981802 4825 consensus_queue.cc:1035] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:09:47.982276 32358 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): Couldn't send request to peer 9ecd907cf5634ac98f3f54f3dbec6849. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b26b6d19699047afaee57d6f2c887664. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:47.982403 4825 consensus_queue.cc:1035] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:47.988898 4825 raft_consensus.cc:2932] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:47.991178 32553 raft_consensus.cc:2932] T b26b6d19699047afaee57d6f2c887664 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:47.991793 32683 raft_consensus.cc:2932] T b26b6d19699047afaee57d6f2c887664 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:47.992058 29873 catalog_manager.cc:5484] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index -1 to 2, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:48.026607 32553 consensus_queue.cc:237] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:48.027954 32683 raft_consensus.cc:1261] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:48.027972 32415 raft_consensus.cc:1261] T 1e03e76474584273ae050595a2286559 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:48.028067 32489 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 -> Peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): Couldn't send request to peer 1d0bb69611af4876a2b3b33c4766e13a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e03e76474584273ae050595a2286559. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:48.028405 5829 consensus_queue.cc:1035] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:48.028581 5552 consensus_queue.cc:1035] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:48.030354 5552 raft_consensus.cc:2932] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:48.030833 32415 raft_consensus.cc:2932] T 1e03e76474584273ae050595a2286559 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:48.032824 32683 raft_consensus.cc:2932] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:48.033380 29870 catalog_manager.cc:5484] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:48.047462 29873 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:48.053161 329 tablet_service.cc:1467] Processing CreateTablet for tablet 6538e323b80d4cf69427756701aa50d2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=af3967040cd045c089f853da9c7c057e]), partition=
I20240501 14:09:48.053414 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6538e323b80d4cf69427756701aa50d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.055470 5666 tablet_bootstrap.cc:492] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:48.055645 30124 tablet_service.cc:1467] Processing CreateTablet for tablet 6538e323b80d4cf69427756701aa50d2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=af3967040cd045c089f853da9c7c057e]), partition=
I20240501 14:09:48.055850 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6538e323b80d4cf69427756701aa50d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.056455 5666 tablet_bootstrap.cc:654] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.057871 5672 tablet_bootstrap.cc:492] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:48.058401 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 6538e323b80d4cf69427756701aa50d2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=af3967040cd045c089f853da9c7c057e]), partition=
I20240501 14:09:48.058636 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6538e323b80d4cf69427756701aa50d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.058776 5672 tablet_bootstrap.cc:654] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.058835 5666 tablet_bootstrap.cc:492] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:48.058916 5666 ts_tablet_manager.cc:1367] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:48.059334 5666 raft_consensus.cc:348] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:48.059444 5666 raft_consensus.cc:374] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.059496 5666 raft_consensus.cc:729] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.059623 5666 consensus_queue.cc:260] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:48.059769 5672 tablet_bootstrap.cc:492] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:48.059844 5672 ts_tablet_manager.cc:1367] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:48.059919 5666 ts_tablet_manager.cc:1398] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:48.060304 5672 raft_consensus.cc:348] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:48.060391 5672 raft_consensus.cc:374] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.060420 5672 raft_consensus.cc:729] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.060542 5672 consensus_queue.cc:260] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:48.060794 5672 ts_tablet_manager.cc:1398] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:48.060837 5548 tablet_bootstrap.cc:492] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:48.061751 5548 tablet_bootstrap.cc:654] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.064891 5548 tablet_bootstrap.cc:492] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:48.064976 5548 ts_tablet_manager.cc:1367] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:48.065323 5548 raft_consensus.cc:348] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:48.065407 5548 raft_consensus.cc:374] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.065440 5548 raft_consensus.cc:729] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.065562 5548 consensus_queue.cc:260] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:48.065820 5548 ts_tablet_manager.cc:1398] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:48.149099 5812 raft_consensus.cc:482] T 14267ff763f5409998f457e14e407ed0 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:48.149222 5812 raft_consensus.cc:504] T 14267ff763f5409998f457e14e407ed0 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:48.149649 5812 leader_election.cc:288] T 14267ff763f5409998f457e14e407ed0 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:48.149927 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14267ff763f5409998f457e14e407ed0" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:48.150089 32683 raft_consensus.cc:2379] T 14267ff763f5409998f457e14e407ed0 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a91029b90bf9415a89a2f70736759ad3 in current term 1: Already voted for candidate 93f2518fbadf4f9991cda93afacb363e in this term.
I20240501 14:09:48.150131 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14267ff763f5409998f457e14e407ed0" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:48.150534 32358 leader_election.cc:302] T 14267ff763f5409998f457e14e407ed0 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3; no voters: 93f2518fbadf4f9991cda93afacb363e, db935501e9104de5b15c8780681d6c38
I20240501 14:09:48.150732 5812 raft_consensus.cc:3037] T 14267ff763f5409998f457e14e407ed0 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.151805 5812 raft_consensus.cc:2726] T 14267ff763f5409998f457e14e407ed0 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:09:48.248893 5130 consensus_queue.cc:1035] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:48.252066 5130 consensus_queue.cc:1035] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:48.257900 4830 consensus_queue.cc:1035] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:48.261364 4830 consensus_queue.cc:1035] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:48.286311 5883 ts_tablet_manager.cc:902] T deee2ab875274d4eaeab84e698241ca8 P 5f8a0cfdfdca4148a75686dc88059cf3: Initiating tablet copy from peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:48.286897 5883 tablet_copy_client.cc:308] T deee2ab875274d4eaeab84e698241ca8 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.129:41387
I20240501 14:09:48.287335 30035 tablet_copy_service.cc:140] P 93f2518fbadf4f9991cda93afacb363e: Received BeginTabletCopySession request for tablet deee2ab875274d4eaeab84e698241ca8 from peer 5f8a0cfdfdca4148a75686dc88059cf3 ({username='slave'} at 127.22.252.134:56355)
I20240501 14:09:48.287421 30035 tablet_copy_service.cc:161] P 93f2518fbadf4f9991cda93afacb363e: Beginning new tablet copy session on tablet deee2ab875274d4eaeab84e698241ca8 from peer 5f8a0cfdfdca4148a75686dc88059cf3 at {username='slave'} at 127.22.252.134:56355: session id = 5f8a0cfdfdca4148a75686dc88059cf3-deee2ab875274d4eaeab84e698241ca8
I20240501 14:09:48.288108 30035 tablet_copy_source_session.cc:205] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:48.289045 5883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deee2ab875274d4eaeab84e698241ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.290958 5883 tablet_copy_client.cc:783] T deee2ab875274d4eaeab84e698241ca8 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:48.291149 5883 tablet_copy_client.cc:647] T deee2ab875274d4eaeab84e698241ca8 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:48.292205 5883 tablet_copy_client.cc:523] T deee2ab875274d4eaeab84e698241ca8 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:48.293421 5883 tablet_bootstrap.cc:492] T deee2ab875274d4eaeab84e698241ca8 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:48.312088 5883 tablet_bootstrap.cc:492] T deee2ab875274d4eaeab84e698241ca8 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:48.312387 5883 tablet_bootstrap.cc:492] T deee2ab875274d4eaeab84e698241ca8 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap complete.
I20240501 14:09:48.312553 5883 ts_tablet_manager.cc:1367] T deee2ab875274d4eaeab84e698241ca8 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.004s
I20240501 14:09:48.312935 5883 raft_consensus.cc:348] T deee2ab875274d4eaeab84e698241ca8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:48.313056 5883 raft_consensus.cc:729] T deee2ab875274d4eaeab84e698241ca8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.313185 5883 consensus_queue.cc:260] T deee2ab875274d4eaeab84e698241ca8 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:48.313479 5883 ts_tablet_manager.cc:1398] T deee2ab875274d4eaeab84e698241ca8 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:48.313947 30035 tablet_copy_service.cc:342] P 93f2518fbadf4f9991cda93afacb363e: Request end of tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-deee2ab875274d4eaeab84e698241ca8 received from {username='slave'} at 127.22.252.134:56355
I20240501 14:09:48.314050 30035 tablet_copy_service.cc:433] P 93f2518fbadf4f9991cda93afacb363e: ending tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-deee2ab875274d4eaeab84e698241ca8 on tablet deee2ab875274d4eaeab84e698241ca8 with peer 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:09:48.314183 5566 raft_consensus.cc:482] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:48.314317 5566 raft_consensus.cc:504] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:48.314816 5566 leader_election.cc:288] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:48.315086 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6538e323b80d4cf69427756701aa50d2" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:48.315183 32683 raft_consensus.cc:2454] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:48.315469 30079 leader_election.cc:302] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d35c16dcb506426993811bb2519d7db3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:48.315627 5566 raft_consensus.cc:2781] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:48.315678 5566 raft_consensus.cc:482] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:48.315713 5566 raft_consensus.cc:3037] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.315728 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6538e323b80d4cf69427756701aa50d2" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:48.315874 349 raft_consensus.cc:2454] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:48.316860 5566 raft_consensus.cc:504] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:48.317231 5566 leader_election.cc:288] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:48.317525 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6538e323b80d4cf69427756701aa50d2" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:48.317646 349 raft_consensus.cc:3037] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.317616 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6538e323b80d4cf69427756701aa50d2" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:48.317728 32683 raft_consensus.cc:3037] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.318544 349 raft_consensus.cc:2454] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:48.318648 32683 raft_consensus.cc:2454] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:48.318847 30081 leader_election.cc:302] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:48.318995 5566 raft_consensus.cc:2781] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:48.319082 5566 raft_consensus.cc:686] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:48.319378 5566 consensus_queue.cc:237] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:48.321221 29864 catalog_manager.cc:5484] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:48.337144 32533 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:48.338125 29995 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:48.347664 30388 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:48.348014 32401 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:48.350296 329 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:48.353396 30256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:48.363425 29864 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:48.370579 32664 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:48.375058 32533 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf197259d9648b5a742dcbb74e7f89b (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b95e6b33a5404ce981d2d09cf6e50f40]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:48.375375 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf197259d9648b5a742dcbb74e7f89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.376356 29995 tablet_service.cc:1467] Processing CreateTablet for tablet c8c532b8351d4335825480b9d8ab1725 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b95e6b33a5404ce981d2d09cf6e50f40]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:48.376405 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 89d3e17f11a4460fa0d0a9a0b8342a66 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b95e6b33a5404ce981d2d09cf6e50f40]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:48.376600 29995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c532b8351d4335825480b9d8ab1725. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.377199 329 tablet_service.cc:1467] Processing CreateTablet for tablet c8c532b8351d4335825480b9d8ab1725 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b95e6b33a5404ce981d2d09cf6e50f40]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:48.377465 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c532b8351d4335825480b9d8ab1725. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.377921 30255 tablet_service.cc:1467] Processing CreateTablet for tablet c8c532b8351d4335825480b9d8ab1725 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b95e6b33a5404ce981d2d09cf6e50f40]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:48.378146 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c532b8351d4335825480b9d8ab1725. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.378922 30256 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf197259d9648b5a742dcbb74e7f89b (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b95e6b33a5404ce981d2d09cf6e50f40]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:48.379141 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf197259d9648b5a742dcbb74e7f89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.379630 5666 tablet_bootstrap.cc:492] T c8c532b8351d4335825480b9d8ab1725 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:48.380512 32401 tablet_service.cc:1467] Processing CreateTablet for tablet 89d3e17f11a4460fa0d0a9a0b8342a66 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b95e6b33a5404ce981d2d09cf6e50f40]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:48.380625 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89d3e17f11a4460fa0d0a9a0b8342a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.380753 32401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89d3e17f11a4460fa0d0a9a0b8342a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.381103 30257 tablet_service.cc:1467] Processing CreateTablet for tablet 89d3e17f11a4460fa0d0a9a0b8342a66 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b95e6b33a5404ce981d2d09cf6e50f40]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:48.381301 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89d3e17f11a4460fa0d0a9a0b8342a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.383569 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf197259d9648b5a742dcbb74e7f89b (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b95e6b33a5404ce981d2d09cf6e50f40]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:48.383810 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf197259d9648b5a742dcbb74e7f89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.384253 5666 tablet_bootstrap.cc:654] T c8c532b8351d4335825480b9d8ab1725 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.386796 5904 tablet_bootstrap.cc:492] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:48.387445 5908 tablet_bootstrap.cc:492] T 89d3e17f11a4460fa0d0a9a0b8342a66 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:48.391574 30124 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:48.394656 5907 tablet_bootstrap.cc:492] T c8c532b8351d4335825480b9d8ab1725 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:48.395145 5908 tablet_bootstrap.cc:654] T 89d3e17f11a4460fa0d0a9a0b8342a66 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.395648 5666 tablet_bootstrap.cc:492] T c8c532b8351d4335825480b9d8ab1725 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:48.395746 5666 ts_tablet_manager.cc:1367] T c8c532b8351d4335825480b9d8ab1725 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.001s
I20240501 14:09:48.396209 5666 raft_consensus.cc:348] T c8c532b8351d4335825480b9d8ab1725 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:48.396342 5666 raft_consensus.cc:374] T c8c532b8351d4335825480b9d8ab1725 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.396378 5666 raft_consensus.cc:729] T c8c532b8351d4335825480b9d8ab1725 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.396492 5666 consensus_queue.cc:260] T c8c532b8351d4335825480b9d8ab1725 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:48.396773 5666 ts_tablet_manager.cc:1398] T c8c532b8351d4335825480b9d8ab1725 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:48.396792 5910 tablet_bootstrap.cc:492] T 9cf197259d9648b5a742dcbb74e7f89b P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:48.397878 5907 tablet_bootstrap.cc:654] T c8c532b8351d4335825480b9d8ab1725 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.397878 5904 tablet_bootstrap.cc:654] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.399500 5910 tablet_bootstrap.cc:654] T 9cf197259d9648b5a742dcbb74e7f89b P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.404157 5907 tablet_bootstrap.cc:492] T c8c532b8351d4335825480b9d8ab1725 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:48.404343 5907 ts_tablet_manager.cc:1367] T c8c532b8351d4335825480b9d8ab1725 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20240501 14:09:48.404349 5910 tablet_bootstrap.cc:492] T 9cf197259d9648b5a742dcbb74e7f89b P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:48.404445 5910 ts_tablet_manager.cc:1367] T 9cf197259d9648b5a742dcbb74e7f89b P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:48.405033 5910 raft_consensus.cc:348] T 9cf197259d9648b5a742dcbb74e7f89b P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:48.405184 5910 raft_consensus.cc:374] T 9cf197259d9648b5a742dcbb74e7f89b P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.405251 5910 raft_consensus.cc:729] T 9cf197259d9648b5a742dcbb74e7f89b P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.405421 5910 consensus_queue.cc:260] T 9cf197259d9648b5a742dcbb74e7f89b P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:48.405649 5908 tablet_bootstrap.cc:492] T 89d3e17f11a4460fa0d0a9a0b8342a66 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:48.405772 5908 ts_tablet_manager.cc:1367] T 89d3e17f11a4460fa0d0a9a0b8342a66 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 14:09:48.405854 5910 ts_tablet_manager.cc:1398] T 9cf197259d9648b5a742dcbb74e7f89b P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:48.406594 5907 raft_consensus.cc:348] T c8c532b8351d4335825480b9d8ab1725 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:48.406740 5907 raft_consensus.cc:374] T c8c532b8351d4335825480b9d8ab1725 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.406785 5907 raft_consensus.cc:729] T c8c532b8351d4335825480b9d8ab1725 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.406914 5907 consensus_queue.cc:260] T c8c532b8351d4335825480b9d8ab1725 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:48.407331 5907 ts_tablet_manager.cc:1398] T c8c532b8351d4335825480b9d8ab1725 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:48.406313 5908 raft_consensus.cc:348] T 89d3e17f11a4460fa0d0a9a0b8342a66 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:48.407758 5908 raft_consensus.cc:374] T 89d3e17f11a4460fa0d0a9a0b8342a66 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.407846 5908 raft_consensus.cc:729] T 89d3e17f11a4460fa0d0a9a0b8342a66 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.408025 5908 consensus_queue.cc:260] T 89d3e17f11a4460fa0d0a9a0b8342a66 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:48.408270 5907 tablet_bootstrap.cc:492] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:48.408684 5908 ts_tablet_manager.cc:1398] T 89d3e17f11a4460fa0d0a9a0b8342a66 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:48.409202 5907 tablet_bootstrap.cc:654] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.412468 5904 tablet_bootstrap.cc:492] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:48.412616 5904 ts_tablet_manager.cc:1367] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240501 14:09:48.413110 5904 raft_consensus.cc:348] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:48.413305 5904 raft_consensus.cc:374] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.413367 5904 raft_consensus.cc:729] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.413518 5904 consensus_queue.cc:260] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:48.414467 5904 ts_tablet_manager.cc:1398] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:48.421133 5906 tablet_bootstrap.cc:492] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:48.422204 5906 tablet_bootstrap.cc:654] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.424794 5906 tablet_bootstrap.cc:492] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:48.424917 5906 ts_tablet_manager.cc:1367] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:48.425320 5906 raft_consensus.cc:348] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:48.425451 5906 raft_consensus.cc:374] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.425501 5906 raft_consensus.cc:729] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.425647 5906 consensus_queue.cc:260] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:48.425765 5905 ts_tablet_manager.cc:902] T 371c725533d84656b7a76c1bc981d038 P 9ecd907cf5634ac98f3f54f3dbec6849: Initiating tablet copy from peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:48.425952 5906 ts_tablet_manager.cc:1398] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:48.426352 5906 tablet_bootstrap.cc:492] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:48.426944 5905 tablet_copy_client.cc:308] T 371c725533d84656b7a76c1bc981d038 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.132:34357
I20240501 14:09:48.427441 30428 tablet_copy_service.cc:140] P d1c217469ae4478ebd49e4ed9f7352d7: Received BeginTabletCopySession request for tablet 371c725533d84656b7a76c1bc981d038 from peer 9ecd907cf5634ac98f3f54f3dbec6849 ({username='slave'} at 127.22.252.136:38643)
I20240501 14:09:48.427533 30428 tablet_copy_service.cc:161] P d1c217469ae4478ebd49e4ed9f7352d7: Beginning new tablet copy session on tablet 371c725533d84656b7a76c1bc981d038 from peer 9ecd907cf5634ac98f3f54f3dbec6849 at {username='slave'} at 127.22.252.136:38643: session id = 9ecd907cf5634ac98f3f54f3dbec6849-371c725533d84656b7a76c1bc981d038
I20240501 14:09:48.428542 30428 tablet_copy_source_session.cc:205] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:48.428998 5906 tablet_bootstrap.cc:654] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.430593 5906 tablet_bootstrap.cc:492] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:48.430706 5906 ts_tablet_manager.cc:1367] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:48.431231 5906 raft_consensus.cc:348] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:48.431367 5906 raft_consensus.cc:374] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.431417 5906 raft_consensus.cc:729] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.431545 5906 consensus_queue.cc:260] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:48.431823 5906 ts_tablet_manager.cc:1398] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:48.431970 4756 consensus_queue.cc:1035] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:48.437636 5788 consensus_queue.cc:1035] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:48.444645 5905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371c725533d84656b7a76c1bc981d038. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.445122 5907 tablet_bootstrap.cc:492] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:48.445209 5907 ts_tablet_manager.cc:1367] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.038s user 0.000s sys 0.002s
I20240501 14:09:48.445618 5907 raft_consensus.cc:348] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:48.445735 5907 raft_consensus.cc:374] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.445772 5907 raft_consensus.cc:729] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.445886 5907 consensus_queue.cc:260] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:48.446183 5907 ts_tablet_manager.cc:1398] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:48.446347 5907 tablet_bootstrap.cc:492] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:48.446821 5905 tablet_copy_client.cc:783] T 371c725533d84656b7a76c1bc981d038 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:48.446964 5905 tablet_copy_client.cc:647] T 371c725533d84656b7a76c1bc981d038 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:48.447197 5907 tablet_bootstrap.cc:654] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.448156 5905 tablet_copy_client.cc:523] T 371c725533d84656b7a76c1bc981d038 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:48.449723 5907 tablet_bootstrap.cc:492] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:48.449836 5907 ts_tablet_manager.cc:1367] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:09:48.450248 5907 raft_consensus.cc:348] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:48.451118 5905 tablet_bootstrap.cc:492] T 371c725533d84656b7a76c1bc981d038 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:48.451807 5907 raft_consensus.cc:374] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.451869 5907 raft_consensus.cc:729] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.452019 5907 consensus_queue.cc:260] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:48.452291 29872 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:48.453378 5907 ts_tablet_manager.cc:1398] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:48.457360 32683 raft_consensus.cc:1261] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:48.457412 349 raft_consensus.cc:1261] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:48.457708 5566 consensus_queue.cc:1035] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:48.458086 5566 consensus_queue.cc:1035] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:48.462253 29864 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet 6538e323b80d4cf69427756701aa50d2 (table rebalancer_dynamic_table [id=af3967040cd045c089f853da9c7c057e]). Expected version 1 got 0
I20240501 14:09:48.462550 29873 catalog_manager.cc:5576] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937) does not have the latest schema for tablet 6538e323b80d4cf69427756701aa50d2 (table rebalancer_dynamic_table [id=af3967040cd045c089f853da9c7c057e]). Expected version 1 got 0
I20240501 14:09:48.466540 29873 catalog_manager.cc:5576] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937) does not have the latest schema for tablet 6538e323b80d4cf69427756701aa50d2 (table rebalancer_dynamic_table [id=af3967040cd045c089f853da9c7c057e]). Expected version 1 got 0
I20240501 14:09:48.467276 5943 tablet.cc:1722] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:48.467631 5943 tablet.cc:1620] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:48.467674 29873 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet 6538e323b80d4cf69427756701aa50d2 (table rebalancer_dynamic_table [id=af3967040cd045c089f853da9c7c057e]). Expected version 1 got 0
I20240501 14:09:48.468052 5942 tablet.cc:1722] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:48.468611 5942 tablet.cc:1620] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:48.470264 5941 tablet.cc:1722] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:48.472801 5941 tablet.cc:1620] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3: MemRowSet was empty: no flush needed.
I20240501 14:09:48.473057 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=af3967040cd045c089f853da9c7c057e] alter complete (version 1)
I20240501 14:09:48.490128 5905 tablet_bootstrap.cc:492] T 371c725533d84656b7a76c1bc981d038 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:48.490381 5905 tablet_bootstrap.cc:492] T 371c725533d84656b7a76c1bc981d038 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap complete.
I20240501 14:09:48.490527 5905 ts_tablet_manager.cc:1367] T 371c725533d84656b7a76c1bc981d038 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.039s user 0.021s sys 0.000s
I20240501 14:09:48.490924 5905 raft_consensus.cc:348] T 371c725533d84656b7a76c1bc981d038 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:48.491050 5905 raft_consensus.cc:729] T 371c725533d84656b7a76c1bc981d038 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.491163 5905 consensus_queue.cc:260] T 371c725533d84656b7a76c1bc981d038 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:48.491480 5905 ts_tablet_manager.cc:1398] T 371c725533d84656b7a76c1bc981d038 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:48.492173 30428 tablet_copy_service.cc:342] P d1c217469ae4478ebd49e4ed9f7352d7: Request end of tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-371c725533d84656b7a76c1bc981d038 received from {username='slave'} at 127.22.252.136:38643
I20240501 14:09:48.492285 30428 tablet_copy_service.cc:433] P d1c217469ae4478ebd49e4ed9f7352d7: ending tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-371c725533d84656b7a76c1bc981d038 on tablet 371c725533d84656b7a76c1bc981d038 with peer 9ecd907cf5634ac98f3f54f3dbec6849
I20240501 14:09:48.498351 5905 ts_tablet_manager.cc:902] T b26b6d19699047afaee57d6f2c887664 P 9ecd907cf5634ac98f3f54f3dbec6849: Initiating tablet copy from peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:48.498720 5905 tablet_copy_client.cc:308] T b26b6d19699047afaee57d6f2c887664 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.133:46265
I20240501 14:09:48.499150 32443 tablet_copy_service.cc:140] P a91029b90bf9415a89a2f70736759ad3: Received BeginTabletCopySession request for tablet b26b6d19699047afaee57d6f2c887664 from peer 9ecd907cf5634ac98f3f54f3dbec6849 ({username='slave'} at 127.22.252.136:38323)
I20240501 14:09:48.499234 32443 tablet_copy_service.cc:161] P a91029b90bf9415a89a2f70736759ad3: Beginning new tablet copy session on tablet b26b6d19699047afaee57d6f2c887664 from peer 9ecd907cf5634ac98f3f54f3dbec6849 at {username='slave'} at 127.22.252.136:38323: session id = 9ecd907cf5634ac98f3f54f3dbec6849-b26b6d19699047afaee57d6f2c887664
I20240501 14:09:48.499922 32443 tablet_copy_source_session.cc:205] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:48.500684 5905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26b6d19699047afaee57d6f2c887664. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.502421 5905 tablet_copy_client.cc:783] T b26b6d19699047afaee57d6f2c887664 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:48.502617 5905 tablet_copy_client.cc:647] T b26b6d19699047afaee57d6f2c887664 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:48.503717 5905 tablet_copy_client.cc:523] T b26b6d19699047afaee57d6f2c887664 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:48.504873 5905 tablet_bootstrap.cc:492] T b26b6d19699047afaee57d6f2c887664 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:48.510581 5959 ts_tablet_manager.cc:902] T 7b3a90c03de64d6891dda267167d1d7b P 1d0bb69611af4876a2b3b33c4766e13a: Initiating tablet copy from peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:48.511211 5959 tablet_copy_client.cc:308] T 7b3a90c03de64d6891dda267167d1d7b P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.129:41387
I20240501 14:09:48.511657 30035 tablet_copy_service.cc:140] P 93f2518fbadf4f9991cda93afacb363e: Received BeginTabletCopySession request for tablet 7b3a90c03de64d6891dda267167d1d7b from peer 1d0bb69611af4876a2b3b33c4766e13a ({username='slave'} at 127.22.252.131:42771)
I20240501 14:09:48.511787 30035 tablet_copy_service.cc:161] P 93f2518fbadf4f9991cda93afacb363e: Beginning new tablet copy session on tablet 7b3a90c03de64d6891dda267167d1d7b from peer 1d0bb69611af4876a2b3b33c4766e13a at {username='slave'} at 127.22.252.131:42771: session id = 1d0bb69611af4876a2b3b33c4766e13a-7b3a90c03de64d6891dda267167d1d7b
I20240501 14:09:48.512789 30035 tablet_copy_source_session.cc:205] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:48.513672 5959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3a90c03de64d6891dda267167d1d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.515906 5959 tablet_copy_client.cc:783] T 7b3a90c03de64d6891dda267167d1d7b P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:48.516080 5959 tablet_copy_client.cc:647] T 7b3a90c03de64d6891dda267167d1d7b P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:48.517167 5959 tablet_copy_client.cc:523] T 7b3a90c03de64d6891dda267167d1d7b P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:48.518761 5959 tablet_bootstrap.cc:492] T 7b3a90c03de64d6891dda267167d1d7b P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:48.525285 5959 tablet_bootstrap.cc:492] T 7b3a90c03de64d6891dda267167d1d7b P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:48.525504 5959 tablet_bootstrap.cc:492] T 7b3a90c03de64d6891dda267167d1d7b P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap complete.
I20240501 14:09:48.525653 5959 ts_tablet_manager.cc:1367] T 7b3a90c03de64d6891dda267167d1d7b P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 14:09:48.525830 5905 tablet_bootstrap.cc:492] T b26b6d19699047afaee57d6f2c887664 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:48.526089 5905 tablet_bootstrap.cc:492] T b26b6d19699047afaee57d6f2c887664 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap complete.
I20240501 14:09:48.526235 5905 ts_tablet_manager.cc:1367] T b26b6d19699047afaee57d6f2c887664 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.003s
I20240501 14:09:48.526110 5959 raft_consensus.cc:348] T 7b3a90c03de64d6891dda267167d1d7b P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:48.526564 5959 raft_consensus.cc:729] T 7b3a90c03de64d6891dda267167d1d7b P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.526641 5905 raft_consensus.cc:348] T b26b6d19699047afaee57d6f2c887664 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:48.526696 5959 consensus_queue.cc:260] T 7b3a90c03de64d6891dda267167d1d7b P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:48.526746 5905 raft_consensus.cc:729] T b26b6d19699047afaee57d6f2c887664 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.526861 5905 consensus_queue.cc:260] T b26b6d19699047afaee57d6f2c887664 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:48.526996 5959 ts_tablet_manager.cc:1398] T 7b3a90c03de64d6891dda267167d1d7b P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:48.527173 5905 ts_tablet_manager.cc:1398] T b26b6d19699047afaee57d6f2c887664 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:48.528178 30035 tablet_copy_service.cc:342] P 93f2518fbadf4f9991cda93afacb363e: Request end of tablet copy session 1d0bb69611af4876a2b3b33c4766e13a-7b3a90c03de64d6891dda267167d1d7b received from {username='slave'} at 127.22.252.131:42771
I20240501 14:09:48.528195 32443 tablet_copy_service.cc:342] P a91029b90bf9415a89a2f70736759ad3: Request end of tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-b26b6d19699047afaee57d6f2c887664 received from {username='slave'} at 127.22.252.136:38323
I20240501 14:09:48.528277 30035 tablet_copy_service.cc:433] P 93f2518fbadf4f9991cda93afacb363e: ending tablet copy session 1d0bb69611af4876a2b3b33c4766e13a-7b3a90c03de64d6891dda267167d1d7b on tablet 7b3a90c03de64d6891dda267167d1d7b with peer 1d0bb69611af4876a2b3b33c4766e13a
I20240501 14:09:48.528277 32443 tablet_copy_service.cc:433] P a91029b90bf9415a89a2f70736759ad3: ending tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-b26b6d19699047afaee57d6f2c887664 on tablet b26b6d19699047afaee57d6f2c887664 with peer 9ecd907cf5634ac98f3f54f3dbec6849
I20240501 14:09:48.529306 5959 ts_tablet_manager.cc:902] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a: Initiating tablet copy from peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:48.529671 5959 tablet_copy_client.cc:308] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.134:33497
I20240501 14:09:48.530720 32572 tablet_copy_service.cc:140] P 5f8a0cfdfdca4148a75686dc88059cf3: Received BeginTabletCopySession request for tablet 1e03e76474584273ae050595a2286559 from peer 1d0bb69611af4876a2b3b33c4766e13a ({username='slave'} at 127.22.252.131:56783)
I20240501 14:09:48.530853 32572 tablet_copy_service.cc:161] P 5f8a0cfdfdca4148a75686dc88059cf3: Beginning new tablet copy session on tablet 1e03e76474584273ae050595a2286559 from peer 1d0bb69611af4876a2b3b33c4766e13a at {username='slave'} at 127.22.252.131:56783: session id = 1d0bb69611af4876a2b3b33c4766e13a-1e03e76474584273ae050595a2286559
I20240501 14:09:48.532074 32572 tablet_copy_source_session.cc:205] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:48.533087 5959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e03e76474584273ae050595a2286559. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.536795 5959 tablet_copy_client.cc:783] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:48.536971 5959 tablet_copy_client.cc:647] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:48.539664 5959 tablet_copy_client.cc:523] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:48.540879 5959 tablet_bootstrap.cc:492] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:48.543182 5130 raft_consensus.cc:482] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:48.543285 5130 raft_consensus.cc:504] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:48.543671 5130 leader_election.cc:288] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:48.544138 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d3e17f11a4460fa0d0a9a0b8342a66" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:48.544296 32415 raft_consensus.cc:2454] T 89d3e17f11a4460fa0d0a9a0b8342a66 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:48.544601 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d3e17f11a4460fa0d0a9a0b8342a66" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:48.544706 30264 raft_consensus.cc:2454] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:48.544987 5995 ts_tablet_manager.cc:902] T b66c71cbccfa4c96bdf2b438938bf56b P d35c16dcb506426993811bb2519d7db3: Initiating tablet copy from peer 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:48.545116 29950 leader_election.cc:302] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:48.545351 5959 tablet_bootstrap.cc:492] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:48.545460 5130 raft_consensus.cc:2781] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:48.545493 5995 tablet_copy_client.cc:308] T b66c71cbccfa4c96bdf2b438938bf56b P d35c16dcb506426993811bb2519d7db3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.136:45937
I20240501 14:09:48.545516 5130 raft_consensus.cc:482] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:48.545576 5130 raft_consensus.cc:3037] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.545677 5959 tablet_bootstrap.cc:492] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap complete.
I20240501 14:09:48.546034 5959 ts_tablet_manager.cc:1367] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20240501 14:09:48.546464 369 tablet_copy_service.cc:140] P 9ecd907cf5634ac98f3f54f3dbec6849: Received BeginTabletCopySession request for tablet b66c71cbccfa4c96bdf2b438938bf56b from peer d35c16dcb506426993811bb2519d7db3 ({username='slave'} at 127.22.252.130:43657)
I20240501 14:09:48.546435 5959 raft_consensus.cc:348] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:48.546540 369 tablet_copy_service.cc:161] P 9ecd907cf5634ac98f3f54f3dbec6849: Beginning new tablet copy session on tablet b66c71cbccfa4c96bdf2b438938bf56b from peer d35c16dcb506426993811bb2519d7db3 at {username='slave'} at 127.22.252.130:43657: session id = d35c16dcb506426993811bb2519d7db3-b66c71cbccfa4c96bdf2b438938bf56b
I20240501 14:09:48.546550 5959 raft_consensus.cc:729] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.546661 5130 raft_consensus.cc:504] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:48.546685 5959 consensus_queue.cc:260] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:48.547000 5959 ts_tablet_manager.cc:1398] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:48.546993 5130 leader_election.cc:288] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:48.547245 369 tablet_copy_source_session.cc:205] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:48.547521 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d3e17f11a4460fa0d0a9a0b8342a66" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:48.547633 32415 raft_consensus.cc:3037] T 89d3e17f11a4460fa0d0a9a0b8342a66 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.547601 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d3e17f11a4460fa0d0a9a0b8342a66" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:48.547701 30264 raft_consensus.cc:3037] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.547854 32572 tablet_copy_service.cc:342] P 5f8a0cfdfdca4148a75686dc88059cf3: Request end of tablet copy session 1d0bb69611af4876a2b3b33c4766e13a-1e03e76474584273ae050595a2286559 received from {username='slave'} at 127.22.252.131:56783
I20240501 14:09:48.547927 32572 tablet_copy_service.cc:433] P 5f8a0cfdfdca4148a75686dc88059cf3: ending tablet copy session 1d0bb69611af4876a2b3b33c4766e13a-1e03e76474584273ae050595a2286559 on tablet 1e03e76474584273ae050595a2286559 with peer 1d0bb69611af4876a2b3b33c4766e13a
I20240501 14:09:48.548563 30264 raft_consensus.cc:2454] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:48.548619 5995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b66c71cbccfa4c96bdf2b438938bf56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.548719 32415 raft_consensus.cc:2454] T 89d3e17f11a4460fa0d0a9a0b8342a66 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:48.549046 29950 leader_election.cc:302] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:48.549324 5730 raft_consensus.cc:482] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:48.549391 5730 raft_consensus.cc:504] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:48.549723 5730 leader_election.cc:288] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:48.549810 5730 raft_consensus.cc:2781] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:48.549866 5730 raft_consensus.cc:686] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:48.549990 5730 consensus_queue.cc:237] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:48.550837 5995 tablet_copy_client.cc:783] T b66c71cbccfa4c96bdf2b438938bf56b P d35c16dcb506426993811bb2519d7db3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:48.550863 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c532b8351d4335825480b9d8ab1725" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:48.550977 5995 tablet_copy_client.cc:647] T b66c71cbccfa4c96bdf2b438938bf56b P d35c16dcb506426993811bb2519d7db3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:48.550976 349 raft_consensus.cc:2454] T c8c532b8351d4335825480b9d8ab1725 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:48.551425 29950 leader_election.cc:302] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:48.551434 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c532b8351d4335825480b9d8ab1725" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:48.551520 30264 raft_consensus.cc:2454] T c8c532b8351d4335825480b9d8ab1725 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:48.551738 5130 raft_consensus.cc:2781] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:48.551784 5130 raft_consensus.cc:482] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:48.551826 5130 raft_consensus.cc:3037] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.552605 29866 catalog_manager.cc:5484] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:48.554090 5130 raft_consensus.cc:504] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:48.554344 5130 leader_election.cc:288] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:48.554646 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c532b8351d4335825480b9d8ab1725" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:48.554721 30277 raft_consensus.cc:3037] T c8c532b8351d4335825480b9d8ab1725 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.555330 30277 raft_consensus.cc:2454] T c8c532b8351d4335825480b9d8ab1725 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:48.555510 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c532b8351d4335825480b9d8ab1725" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:48.555603 349 raft_consensus.cc:3037] T c8c532b8351d4335825480b9d8ab1725 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.556488 349 raft_consensus.cc:2454] T c8c532b8351d4335825480b9d8ab1725 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:48.556566 29950 leader_election.cc:302] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:48.556679 5130 raft_consensus.cc:2781] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:48.556783 5130 raft_consensus.cc:686] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:48.557103 5130 consensus_queue.cc:237] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:48.558601 5995 tablet_copy_client.cc:523] T b66c71cbccfa4c96bdf2b438938bf56b P d35c16dcb506426993811bb2519d7db3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:48.559067 29870 catalog_manager.cc:5484] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:48.561240 5995 tablet_bootstrap.cc:492] T b66c71cbccfa4c96bdf2b438938bf56b P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:48.566478 5995 tablet_bootstrap.cc:492] T b66c71cbccfa4c96bdf2b438938bf56b P d35c16dcb506426993811bb2519d7db3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:48.566722 5995 tablet_bootstrap.cc:492] T b66c71cbccfa4c96bdf2b438938bf56b P d35c16dcb506426993811bb2519d7db3: Bootstrap complete.
I20240501 14:09:48.566874 5995 ts_tablet_manager.cc:1367] T b66c71cbccfa4c96bdf2b438938bf56b P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 14:09:48.567292 5995 raft_consensus.cc:348] T b66c71cbccfa4c96bdf2b438938bf56b P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:48.567399 5995 raft_consensus.cc:729] T b66c71cbccfa4c96bdf2b438938bf56b P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.567518 5995 consensus_queue.cc:260] T b66c71cbccfa4c96bdf2b438938bf56b P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:48.567816 5995 ts_tablet_manager.cc:1398] T b66c71cbccfa4c96bdf2b438938bf56b P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:48.568904 369 tablet_copy_service.cc:342] P 9ecd907cf5634ac98f3f54f3dbec6849: Request end of tablet copy session d35c16dcb506426993811bb2519d7db3-b66c71cbccfa4c96bdf2b438938bf56b received from {username='slave'} at 127.22.252.130:43657
I20240501 14:09:48.568977 369 tablet_copy_service.cc:433] P 9ecd907cf5634ac98f3f54f3dbec6849: ending tablet copy session d35c16dcb506426993811bb2519d7db3-b66c71cbccfa4c96bdf2b438938bf56b on tablet b66c71cbccfa4c96bdf2b438938bf56b with peer d35c16dcb506426993811bb2519d7db3
I20240501 14:09:48.574568 29873 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:48.586709 5941 tablet.cc:1722] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:48.586982 5941 tablet.cc:1620] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3: MemRowSet was empty: no flush needed.
I20240501 14:09:48.587049 5943 tablet.cc:1722] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:48.587345 5943 tablet.cc:1620] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:48.588281 5942 tablet.cc:1722] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:48.588563 5942 tablet.cc:1620] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:48.592308 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=af3967040cd045c089f853da9c7c057e] alter complete (version 2)
I20240501 14:09:48.675666 5552 raft_consensus.cc:482] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:48.675772 5552 raft_consensus.cc:504] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:48.676167 5552 leader_election.cc:288] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:48.676510 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf197259d9648b5a742dcbb74e7f89b" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:48.676551 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf197259d9648b5a742dcbb74e7f89b" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:48.676654 30406 raft_consensus.cc:2454] T 9cf197259d9648b5a742dcbb74e7f89b P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:48.676676 30277 raft_consensus.cc:2454] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:48.676961 32489 leader_election.cc:302] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 5f8a0cfdfdca4148a75686dc88059cf3; no voters:
I20240501 14:09:48.677116 5552 raft_consensus.cc:2781] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:48.677215 5552 raft_consensus.cc:482] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:48.677263 5552 raft_consensus.cc:3037] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.678309 5552 raft_consensus.cc:504] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:48.678618 5552 leader_election.cc:288] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:48.678920 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf197259d9648b5a742dcbb74e7f89b" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:48.679042 30277 raft_consensus.cc:3037] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.679140 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf197259d9648b5a742dcbb74e7f89b" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:48.679252 30406 raft_consensus.cc:3037] T 9cf197259d9648b5a742dcbb74e7f89b P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.680071 30277 raft_consensus.cc:2454] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:48.680272 30406 raft_consensus.cc:2454] T 9cf197259d9648b5a742dcbb74e7f89b P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:48.680408 32489 leader_election.cc:302] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 5f8a0cfdfdca4148a75686dc88059cf3; no voters:
I20240501 14:09:48.680542 5552 raft_consensus.cc:2781] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:48.680682 5552 raft_consensus.cc:686] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:09:48.680805 5552 consensus_queue.cc:237] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:48.682796 29872 catalog_manager.cc:5484] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:48.695740 29872 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:48.698872 32533 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:48.701270 30255 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:48.706084 30124 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:48.708714 29993 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:48.710016 32401 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:48.713761 30388 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:48.713941 5941 tablet.cc:1722] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:48.718971 5942 tablet.cc:1722] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:48.722436 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=af3967040cd045c089f853da9c7c057e] alter complete (version 3)
I20240501 14:09:48.729322 5943 tablet.cc:1722] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:48.736450 329 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:48.747144 32664 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:48.774482 32553 raft_consensus.cc:1203] T deee2ab875274d4eaeab84e698241ca8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:48.840775 29868 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:48.840934 29868 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:48.843828 29868 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 6538e323b80d4cf69427756701aa50d2
I20240501 14:09:48.844544 32664 tablet_service.cc:1514] Processing DeleteTablet for tablet 6538e323b80d4cf69427756701aa50d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:48 UTC) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:48.844657 30124 tablet_service.cc:1514] Processing DeleteTablet for tablet 6538e323b80d4cf69427756701aa50d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:48 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:48.844887 329 tablet_service.cc:1514] Processing DeleteTablet for tablet 6538e323b80d4cf69427756701aa50d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:48 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:48.845130 6060 tablet_replica.cc:330] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:48.845320 6060 raft_consensus.cc:2229] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:48.845487 6060 raft_consensus.cc:2258] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:48.845796 6062 tablet_replica.cc:330] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:48.846032 6062 raft_consensus.cc:2229] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:48.846206 6062 raft_consensus.cc:2258] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:48.846230 6060 ts_tablet_manager.cc:1875] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:48.847277 6062 ts_tablet_manager.cc:1875] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:48.850857 6060 ts_tablet_manager.cc:1888] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:48.850996 6060 log.cc:1196] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/6538e323b80d4cf69427756701aa50d2
I20240501 14:09:48.851349 6060 ts_tablet_manager.cc:1909] T 6538e323b80d4cf69427756701aa50d2 P db935501e9104de5b15c8780681d6c38: Deleting consensus metadata
I20240501 14:09:48.851734 6062 ts_tablet_manager.cc:1888] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:48.851912 6062 log.cc:1196] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/6538e323b80d4cf69427756701aa50d2
I20240501 14:09:48.852197 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet 6538e323b80d4cf69427756701aa50d2 (table rebalancer_dynamic_table_new_name [id=af3967040cd045c089f853da9c7c057e]) successfully deleted
I20240501 14:09:48.852331 6062 ts_tablet_manager.cc:1909] T 6538e323b80d4cf69427756701aa50d2 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:09:48.853014 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 6538e323b80d4cf69427756701aa50d2 (table rebalancer_dynamic_table_new_name [id=af3967040cd045c089f853da9c7c057e]) successfully deleted
I20240501 14:09:48.857098 6061 tablet_replica.cc:330] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:48.857318 6061 raft_consensus.cc:2229] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:48.857575 6061 raft_consensus.cc:2258] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:48.858140 6061 ts_tablet_manager.cc:1875] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:48.863497 6061 ts_tablet_manager.cc:1888] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:48.864202 6061 log.cc:1196] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/6538e323b80d4cf69427756701aa50d2
I20240501 14:09:48.864598 6061 ts_tablet_manager.cc:1909] T 6538e323b80d4cf69427756701aa50d2 P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:48.865322 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 6538e323b80d4cf69427756701aa50d2 (table rebalancer_dynamic_table_new_name [id=af3967040cd045c089f853da9c7c057e]) successfully deleted
I20240501 14:09:48.883632 30255 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:48.885738 32533 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:48.890630 329 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:48.895164 29993 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:48.896389 30388 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:48.906778 30124 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:48.917481 30277 raft_consensus.cc:1203] T 7b3a90c03de64d6891dda267167d1d7b P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:48.920256 32401 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:48.930861 5130 consensus_queue.cc:1035] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:48.935122 6075 consensus_queue.cc:1035] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 14:09:48.947224 348 raft_consensus.cc:1203] T 371c725533d84656b7a76c1bc981d038 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:48.948051 32664 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:48.948833 29873 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:48.956653 32401 tablet_service.cc:1467] Processing CreateTablet for tablet 4cd842ef119e4a48966766094aeb0b94 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3805d326c7e843e1b557de84dc07d6ce]), partition=
I20240501 14:09:48.956903 32401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd842ef119e4a48966766094aeb0b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.957624 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 4cd842ef119e4a48966766094aeb0b94 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3805d326c7e843e1b557de84dc07d6ce]), partition=
I20240501 14:09:48.957827 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd842ef119e4a48966766094aeb0b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.960343 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 4cd842ef119e4a48966766094aeb0b94 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3805d326c7e843e1b557de84dc07d6ce]), partition=
I20240501 14:09:48.960580 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd842ef119e4a48966766094aeb0b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.960736 6082 tablet_bootstrap.cc:492] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:48.961601 6082 tablet_bootstrap.cc:654] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.961930 6081 tablet_bootstrap.cc:492] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:48.962708 6082 tablet_bootstrap.cc:492] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:48.962792 6082 ts_tablet_manager.cc:1367] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:48.962850 6081 tablet_bootstrap.cc:654] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.963176 6082 raft_consensus.cc:348] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:48.963248 6082 raft_consensus.cc:374] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.963287 6082 raft_consensus.cc:729] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.963382 6082 consensus_queue.cc:260] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:48.963658 6082 ts_tablet_manager.cc:1398] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:48.964206 6084 tablet_bootstrap.cc:492] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:48.964738 6081 tablet_bootstrap.cc:492] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:48.964862 6081 ts_tablet_manager.cc:1367] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:48.964923 6084 tablet_bootstrap.cc:654] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.965345 6081 raft_consensus.cc:348] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:48.965456 6081 raft_consensus.cc:374] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.965519 6081 raft_consensus.cc:729] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.965670 6081 consensus_queue.cc:260] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:48.965939 6084 tablet_bootstrap.cc:492] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:48.966054 6081 ts_tablet_manager.cc:1398] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:48.966066 6084 ts_tablet_manager.cc:1367] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:48.966444 6084 raft_consensus.cc:348] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:48.966569 6084 raft_consensus.cc:374] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.966614 6084 raft_consensus.cc:729] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.966779 6084 consensus_queue.cc:260] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:48.967867 6084 ts_tablet_manager.cc:1398] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:49.003706 5070 raft_consensus.cc:482] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:49.003803 5070 raft_consensus.cc:504] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:49.004388 5070 leader_election.cc:288] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:49.004547 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd842ef119e4a48966766094aeb0b94" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:49.004685 32415 raft_consensus.cc:2454] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:49.004967 32620 leader_election.cc:302] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:49.005172 5070 raft_consensus.cc:2781] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:49.005229 5070 raft_consensus.cc:482] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:49.005272 5070 raft_consensus.cc:3037] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.005318 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd842ef119e4a48966766094aeb0b94" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:49.005518 30406 raft_consensus.cc:2454] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:49.006282 5070 raft_consensus.cc:504] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:49.006647 5070 leader_election.cc:288] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Requested vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:49.006925 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd842ef119e4a48966766094aeb0b94" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:49.007027 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd842ef119e4a48966766094aeb0b94" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:49.007076 32415 raft_consensus.cc:3037] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.007134 30406 raft_consensus.cc:3037] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.008067 32415 raft_consensus.cc:2454] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:49.008080 30406 raft_consensus.cc:2454] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:49.008368 32618 leader_election.cc:302] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:49.008543 5070 raft_consensus.cc:2781] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:49.008628 5070 raft_consensus.cc:686] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Becoming Leader. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
I20240501 14:09:49.008754 5070 consensus_queue.cc:237] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:49.010895 29870 catalog_manager.cc:5484] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db935501e9104de5b15c8780681d6c38 (127.22.252.135). New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:49.032716 5730 consensus_queue.cc:1035] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:49.036666 5730 consensus_queue.cc:1035] T c8c532b8351d4335825480b9d8ab1725 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:49.057338 30264 raft_consensus.cc:1203] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:49.077806 348 raft_consensus.cc:1203] T b26b6d19699047afaee57d6f2c887664 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:49.085793 5552 consensus_queue.cc:1035] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:49.091472 6004 consensus_queue.cc:1035] T 9cf197259d9648b5a742dcbb74e7f89b P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:49.105093 29993 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:49.109994 30388 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:49.110108 30255 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:49.112711 32533 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:49.120478 29873 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:49.125598 30406 raft_consensus.cc:1261] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:49.125986 5070 consensus_queue.cc:1035] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:49.126112 32416 raft_consensus.cc:1261] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:49.126497 5070 consensus_queue.cc:1035] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:49.129751 29872 catalog_manager.cc:5576] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357) does not have the latest schema for tablet 4cd842ef119e4a48966766094aeb0b94 (table rebalancer_dynamic_table [id=3805d326c7e843e1b557de84dc07d6ce]). Expected version 1 got 0
I20240501 14:09:49.129943 5943 tablet.cc:1722] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:49.130268 5943 tablet.cc:1620] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:49.131374 29873 catalog_manager.cc:5576] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265) does not have the latest schema for tablet 4cd842ef119e4a48966766094aeb0b94 (table rebalancer_dynamic_table [id=3805d326c7e843e1b557de84dc07d6ce]). Expected version 1 got 0
I20240501 14:09:49.132326 32401 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:49.134419 329 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:49.136303 30124 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:49.137127 29873 catalog_manager.cc:5576] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265) does not have the latest schema for tablet 4cd842ef119e4a48966766094aeb0b94 (table rebalancer_dynamic_table [id=3805d326c7e843e1b557de84dc07d6ce]). Expected version 1 got 0
I20240501 14:09:49.138895 29872 catalog_manager.cc:5576] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357) does not have the latest schema for tablet 4cd842ef119e4a48966766094aeb0b94 (table rebalancer_dynamic_table [id=3805d326c7e843e1b557de84dc07d6ce]). Expected version 1 got 0
I20240501 14:09:49.142275 29873 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=3805d326c7e843e1b557de84dc07d6ce] alter complete (version 1)
I20240501 14:09:49.148456 32664 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:49.151697 6151 tablet.cc:1722] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:49.152045 6151 tablet.cc:1620] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:49.153095 30146 raft_consensus.cc:1203] T b66c71cbccfa4c96bdf2b438938bf56b P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:49.170876 6152 tablet.cc:1722] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:49.171193 6152 tablet.cc:1620] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7: MemRowSet was empty: no flush needed.
I20240501 14:09:49.205181 29873 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:49.215785 32401 tablet_service.cc:1467] Processing CreateTablet for tablet 91e27086d98443aa9acec74d6283cd75 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=5cc0b546e462440f9284a6628020d724]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.215890 30124 tablet_service.cc:1467] Processing CreateTablet for tablet d8430bd8cb7549f0aa4ede9a3d1ffe6e (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=5cc0b546e462440f9284a6628020d724]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.215988 30125 tablet_service.cc:1467] Processing CreateTablet for tablet 91e27086d98443aa9acec74d6283cd75 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=5cc0b546e462440f9284a6628020d724]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.216111 32401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e27086d98443aa9acec74d6283cd75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.216135 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8430bd8cb7549f0aa4ede9a3d1ffe6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.216223 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e27086d98443aa9acec74d6283cd75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.218430 30126 tablet_service.cc:1467] Processing CreateTablet for tablet 745976e20bdc47268fad141ceccd1a81 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=5cc0b546e462440f9284a6628020d724]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.218492 32533 tablet_service.cc:1467] Processing CreateTablet for tablet 91e27086d98443aa9acec74d6283cd75 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=5cc0b546e462440f9284a6628020d724]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.218627 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745976e20bdc47268fad141ceccd1a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.218775 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e27086d98443aa9acec74d6283cd75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.220386 6081 tablet_bootstrap.cc:492] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:49.220561 329 tablet_service.cc:1467] Processing CreateTablet for tablet 745976e20bdc47268fad141ceccd1a81 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=5cc0b546e462440f9284a6628020d724]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.220818 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745976e20bdc47268fad141ceccd1a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.221307 6081 tablet_bootstrap.cc:654] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.221339 32664 tablet_service.cc:1467] Processing CreateTablet for tablet d8430bd8cb7549f0aa4ede9a3d1ffe6e (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=5cc0b546e462440f9284a6628020d724]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.221484 30387 tablet_service.cc:1467] Processing CreateTablet for tablet 745976e20bdc47268fad141ceccd1a81 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=5cc0b546e462440f9284a6628020d724]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.221587 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8430bd8cb7549f0aa4ede9a3d1ffe6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.221730 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745976e20bdc47268fad141ceccd1a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.223717 6155 tablet_bootstrap.cc:492] T 745976e20bdc47268fad141ceccd1a81 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:49.223800 6084 tablet_bootstrap.cc:492] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:49.223811 6153 tablet_bootstrap.cc:492] T 91e27086d98443aa9acec74d6283cd75 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:49.223913 30388 tablet_service.cc:1467] Processing CreateTablet for tablet d8430bd8cb7549f0aa4ede9a3d1ffe6e (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=5cc0b546e462440f9284a6628020d724]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.224112 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8430bd8cb7549f0aa4ede9a3d1ffe6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.224608 6153 tablet_bootstrap.cc:654] T 91e27086d98443aa9acec74d6283cd75 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.224634 6155 tablet_bootstrap.cc:654] T 745976e20bdc47268fad141ceccd1a81 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.224782 6084 tablet_bootstrap.cc:654] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.225806 6084 tablet_bootstrap.cc:492] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:49.225888 6084 ts_tablet_manager.cc:1367] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:49.226262 6082 tablet_bootstrap.cc:492] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:49.226347 6084 raft_consensus.cc:348] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:49.226426 6084 raft_consensus.cc:374] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.226446 6084 raft_consensus.cc:729] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.226547 6084 consensus_queue.cc:260] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:49.226758 6159 tablet_bootstrap.cc:492] T 91e27086d98443aa9acec74d6283cd75 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:49.226822 6084 ts_tablet_manager.cc:1398] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:49.226994 6084 tablet_bootstrap.cc:492] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:49.227224 6082 tablet_bootstrap.cc:654] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.227660 6159 tablet_bootstrap.cc:654] T 91e27086d98443aa9acec74d6283cd75 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.228055 6084 tablet_bootstrap.cc:654] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.230597 6082 tablet_bootstrap.cc:492] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:49.230690 6082 ts_tablet_manager.cc:1367] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:49.231145 6082 raft_consensus.cc:348] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:49.231276 6082 raft_consensus.cc:374] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.231310 6082 raft_consensus.cc:729] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.231454 6082 consensus_queue.cc:260] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:49.231755 6082 ts_tablet_manager.cc:1398] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:49.232973 6153 tablet_bootstrap.cc:492] T 91e27086d98443aa9acec74d6283cd75 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:49.233075 6153 ts_tablet_manager.cc:1367] T 91e27086d98443aa9acec74d6283cd75 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:49.233551 6084 tablet_bootstrap.cc:492] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:49.233572 6153 raft_consensus.cc:348] T 91e27086d98443aa9acec74d6283cd75 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:49.233621 6155 tablet_bootstrap.cc:492] T 745976e20bdc47268fad141ceccd1a81 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:49.233639 6084 ts_tablet_manager.cc:1367] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 14:09:49.233669 6153 raft_consensus.cc:374] T 91e27086d98443aa9acec74d6283cd75 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.233700 6153 raft_consensus.cc:729] T 91e27086d98443aa9acec74d6283cd75 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.233714 6155 ts_tablet_manager.cc:1367] T 745976e20bdc47268fad141ceccd1a81 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 14:09:49.233822 6153 consensus_queue.cc:260] T 91e27086d98443aa9acec74d6283cd75 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:49.234092 6153 ts_tablet_manager.cc:1398] T 91e27086d98443aa9acec74d6283cd75 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:49.234057 6084 raft_consensus.cc:348] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:49.234148 6084 raft_consensus.cc:374] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.234184 6084 raft_consensus.cc:729] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.234198 6155 raft_consensus.cc:348] T 745976e20bdc47268fad141ceccd1a81 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:49.234310 6155 raft_consensus.cc:374] T 745976e20bdc47268fad141ceccd1a81 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.234347 6155 raft_consensus.cc:729] T 745976e20bdc47268fad141ceccd1a81 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.234316 6084 consensus_queue.cc:260] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:49.234464 6155 consensus_queue.cc:260] T 745976e20bdc47268fad141ceccd1a81 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:49.234561 6084 ts_tablet_manager.cc:1398] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:49.234719 6155 ts_tablet_manager.cc:1398] T 745976e20bdc47268fad141ceccd1a81 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:49.235867 6153 tablet_bootstrap.cc:492] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:49.238178 6153 tablet_bootstrap.cc:654] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.241031 6159 tablet_bootstrap.cc:492] T 91e27086d98443aa9acec74d6283cd75 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:49.241031 6081 tablet_bootstrap.cc:492] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:49.241128 6081 ts_tablet_manager.cc:1367] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240501 14:09:49.241281 6159 ts_tablet_manager.cc:1367] T 91e27086d98443aa9acec74d6283cd75 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 14:09:49.241619 6081 raft_consensus.cc:348] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:49.241744 6081 raft_consensus.cc:374] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.241786 6081 raft_consensus.cc:729] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.241933 6081 consensus_queue.cc:260] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:49.241935 6159 raft_consensus.cc:348] T 91e27086d98443aa9acec74d6283cd75 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:49.242039 6159 raft_consensus.cc:374] T 91e27086d98443aa9acec74d6283cd75 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.242259 6081 ts_tablet_manager.cc:1398] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:49.243795 6159 raft_consensus.cc:729] T 91e27086d98443aa9acec74d6283cd75 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.244024 6159 consensus_queue.cc:260] T 91e27086d98443aa9acec74d6283cd75 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:49.244505 6159 ts_tablet_manager.cc:1398] T 91e27086d98443aa9acec74d6283cd75 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:49.245256 6153 tablet_bootstrap.cc:492] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:49.245349 6153 ts_tablet_manager.cc:1367] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:49.245774 6153 raft_consensus.cc:348] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:49.245862 6153 raft_consensus.cc:374] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.245887 6153 raft_consensus.cc:729] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.245996 6153 consensus_queue.cc:260] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:49.246230 6153 ts_tablet_manager.cc:1398] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:49.246623 6153 tablet_bootstrap.cc:492] T 745976e20bdc47268fad141ceccd1a81 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:49.247512 6153 tablet_bootstrap.cc:654] T 745976e20bdc47268fad141ceccd1a81 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.249008 6153 tablet_bootstrap.cc:492] T 745976e20bdc47268fad141ceccd1a81 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:49.249112 6153 ts_tablet_manager.cc:1367] T 745976e20bdc47268fad141ceccd1a81 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:49.249529 6153 raft_consensus.cc:348] T 745976e20bdc47268fad141ceccd1a81 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:49.249636 6153 raft_consensus.cc:374] T 745976e20bdc47268fad141ceccd1a81 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.249675 6153 raft_consensus.cc:729] T 745976e20bdc47268fad141ceccd1a81 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.249781 6153 consensus_queue.cc:260] T 745976e20bdc47268fad141ceccd1a81 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:49.249898 29870 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:49.250056 6153 ts_tablet_manager.cc:1398] T 745976e20bdc47268fad141ceccd1a81 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:49.255725 5943 tablet.cc:1722] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:49.256055 5943 tablet.cc:1620] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:49.256212 6151 tablet.cc:1722] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:49.256533 6151 tablet.cc:1620] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:49.257084 6152 tablet.cc:1722] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:49.257344 6152 tablet.cc:1620] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7: MemRowSet was empty: no flush needed.
I20240501 14:09:49.259155 6179 raft_consensus.cc:482] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:49.259290 6179 raft_consensus.cc:504] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:49.260165 30408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8430bd8cb7549f0aa4ede9a3d1ffe6e" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:49.260316 30408 raft_consensus.cc:2454] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:49.260598 32618 leader_election.cc:302] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:49.260857 5070 raft_consensus.cc:2781] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:49.260921 5070 raft_consensus.cc:482] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:49.260968 5070 raft_consensus.cc:3037] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.261723 5070 raft_consensus.cc:504] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:49.262087 5070 leader_election.cc:288] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:49.262446 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8430bd8cb7549f0aa4ede9a3d1ffe6e" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:49.262557 30146 raft_consensus.cc:3037] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.262897 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8430bd8cb7549f0aa4ede9a3d1ffe6e" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:49.263000 30406 raft_consensus.cc:3037] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.263260 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8430bd8cb7549f0aa4ede9a3d1ffe6e" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:49.263446 29873 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=3805d326c7e843e1b557de84dc07d6ce] alter complete (version 2)
I20240501 14:09:49.263594 30146 raft_consensus.cc:2454] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:49.263965 30406 raft_consensus.cc:2454] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:49.264057 32621 leader_election.cc:302] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d35c16dcb506426993811bb2519d7db3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:49.264346 5070 raft_consensus.cc:2781] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:49.264425 5070 raft_consensus.cc:686] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Becoming Leader. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
I20240501 14:09:49.264539 5070 consensus_queue.cc:237] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:49.265537 6179 leader_election.cc:288] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:49.266705 29872 catalog_manager.cc:5484] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db935501e9104de5b15c8780681d6c38 (127.22.252.135). New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:49.267747 30406 consensus_queue.cc:237] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:49.269364 30264 raft_consensus.cc:1261] T 371c725533d84656b7a76c1bc981d038 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:49.269382 348 raft_consensus.cc:1261] T 371c725533d84656b7a76c1bc981d038 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:49.269744 5618 consensus_queue.cc:1035] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:49.269873 6028 consensus_queue.cc:1035] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:49.271744 6185 raft_consensus.cc:2932] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:49.272145 348 raft_consensus.cc:2932] T 371c725533d84656b7a76c1bc981d038 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:49.272322 30264 raft_consensus.cc:2932] T 371c725533d84656b7a76c1bc981d038 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:49.275171 29872 catalog_manager.cc:5484] T 371c725533d84656b7a76c1bc981d038 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:49.275314 6133 raft_consensus.cc:482] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:49.275398 6133 raft_consensus.cc:504] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:49.275720 6133 leader_election.cc:288] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:49.276820 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e27086d98443aa9acec74d6283cd75" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:49.276892 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e27086d98443aa9acec74d6283cd75" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:49.277009 30145 raft_consensus.cc:2454] T 91e27086d98443aa9acec74d6283cd75 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:49.277007 32553 raft_consensus.cc:2454] T 91e27086d98443aa9acec74d6283cd75 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:49.277794 32359 leader_election.cc:302] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:49.278129 30126 tablet_service.cc:1514] Processing DeleteTablet for tablet 371c725533d84656b7a76c1bc981d038 with delete_type TABLET_DATA_TOMBSTONED (TS d35c16dcb506426993811bb2519d7db3 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:49.278438 6133 raft_consensus.cc:2781] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:49.278499 6133 raft_consensus.cc:482] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:49.278537 6133 raft_consensus.cc:3037] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.278584 6061 tablet_replica.cc:330] T 371c725533d84656b7a76c1bc981d038 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:49.278726 6061 raft_consensus.cc:2229] T 371c725533d84656b7a76c1bc981d038 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:49.278854 6061 raft_consensus.cc:2258] T 371c725533d84656b7a76c1bc981d038 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:49.279384 6061 ts_tablet_manager.cc:1875] T 371c725533d84656b7a76c1bc981d038 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:49.279597 6133 raft_consensus.cc:504] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:49.279919 6133 leader_election.cc:288] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:49.280351 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e27086d98443aa9acec74d6283cd75" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:49.280444 30145 raft_consensus.cc:3037] T 91e27086d98443aa9acec74d6283cd75 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.280761 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e27086d98443aa9acec74d6283cd75" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:49.280869 32554 raft_consensus.cc:3037] T 91e27086d98443aa9acec74d6283cd75 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.281378 30145 raft_consensus.cc:2454] T 91e27086d98443aa9acec74d6283cd75 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:49.281847 32359 leader_election.cc:302] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:49.281894 32554 raft_consensus.cc:2454] T 91e27086d98443aa9acec74d6283cd75 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:49.282018 5812 raft_consensus.cc:2781] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:49.282088 5812 raft_consensus.cc:686] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:49.282203 5812 consensus_queue.cc:237] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:49.284415 6061 ts_tablet_manager.cc:1888] T 371c725533d84656b7a76c1bc981d038 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:49.284492 6061 log.cc:1196] T 371c725533d84656b7a76c1bc981d038 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/371c725533d84656b7a76c1bc981d038
I20240501 14:09:49.285624 29870 catalog_manager.cc:5484] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:49.286137 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 371c725533d84656b7a76c1bc981d038 (table rebalance_test_table_15 [id=df0504ee71e643e3a5ec9a20e1037981]) successfully deleted
I20240501 14:09:49.365665 29868 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:49.370965 5943 tablet.cc:1722] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:49.371835 6152 tablet.cc:1722] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:49.373574 6151 tablet.cc:1722] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:49.376641 29864 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=3805d326c7e843e1b557de84dc07d6ce] alter complete (version 3)
I20240501 14:09:49.388206 6185 raft_consensus.cc:482] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:49.388335 6185 raft_consensus.cc:504] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:49.388659 6185 leader_election.cc:288] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:49.388937 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745976e20bdc47268fad141ceccd1a81" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:49.389065 30145 raft_consensus.cc:2454] T 745976e20bdc47268fad141ceccd1a81 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:49.389066 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745976e20bdc47268fad141ceccd1a81" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:49.389182 348 raft_consensus.cc:2454] T 745976e20bdc47268fad141ceccd1a81 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:49.389315 30344 leader_election.cc:302] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:49.389452 6185 raft_consensus.cc:2781] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:49.389508 6185 raft_consensus.cc:482] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:49.389542 6185 raft_consensus.cc:3037] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.390390 6185 raft_consensus.cc:504] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:49.390710 6185 leader_election.cc:288] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:49.390966 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745976e20bdc47268fad141ceccd1a81" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:49.391068 30145 raft_consensus.cc:3037] T 745976e20bdc47268fad141ceccd1a81 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.391191 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745976e20bdc47268fad141ceccd1a81" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:49.391295 348 raft_consensus.cc:3037] T 745976e20bdc47268fad141ceccd1a81 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.391880 30145 raft_consensus.cc:2454] T 745976e20bdc47268fad141ceccd1a81 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:49.392093 348 raft_consensus.cc:2454] T 745976e20bdc47268fad141ceccd1a81 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:49.392360 30343 leader_election.cc:302] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:49.392498 6185 raft_consensus.cc:2781] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:49.392581 6185 raft_consensus.cc:686] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:49.392722 6185 consensus_queue.cc:237] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:49.394731 29868 catalog_manager.cc:5484] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:49.412969 29993 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:49.413667 32533 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:49.414117 30255 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:49.423949 329 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:49.427083 32401 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:49.448323 32664 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:49.452082 30388 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:49.456375 30126 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:49.477962 29868 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:49.478099 29868 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:49.480314 29868 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 4cd842ef119e4a48966766094aeb0b94
I20240501 14:09:49.481664 32401 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cd842ef119e4a48966766094aeb0b94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:49 UTC) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:49.482286 30388 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cd842ef119e4a48966766094aeb0b94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:49 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:49.482357 6240 tablet_replica.cc:330] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:49.482528 6240 raft_consensus.cc:2229] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:49.482661 6240 raft_consensus.cc:2258] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:49.483330 6240 ts_tablet_manager.cc:1875] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:49.483400 32664 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cd842ef119e4a48966766094aeb0b94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:49 UTC) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:49.483944 6241 tablet_replica.cc:330] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:49.484961 6242 tablet_replica.cc:330] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:49.485482 6242 raft_consensus.cc:2229] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:49.485167 6241 raft_consensus.cc:2229] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:49.485720 6241 raft_consensus.cc:2258] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:49.485962 6242 raft_consensus.cc:2258] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:49.486902 6242 ts_tablet_manager.cc:1875] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:49.488281 6241 ts_tablet_manager.cc:1875] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:49.489054 6240 ts_tablet_manager.cc:1888] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:49.489166 6240 log.cc:1196] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/4cd842ef119e4a48966766094aeb0b94
I20240501 14:09:49.489158 6242 ts_tablet_manager.cc:1888] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:49.489218 6242 log.cc:1196] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/4cd842ef119e4a48966766094aeb0b94
I20240501 14:09:49.489492 6242 ts_tablet_manager.cc:1909] T 4cd842ef119e4a48966766094aeb0b94 P db935501e9104de5b15c8780681d6c38: Deleting consensus metadata
I20240501 14:09:49.489492 6240 ts_tablet_manager.cc:1909] T 4cd842ef119e4a48966766094aeb0b94 P a91029b90bf9415a89a2f70736759ad3: Deleting consensus metadata
I20240501 14:09:49.490125 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet 4cd842ef119e4a48966766094aeb0b94 (table rebalancer_dynamic_table_new_name [id=3805d326c7e843e1b557de84dc07d6ce]) successfully deleted
I20240501 14:09:49.490159 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 4cd842ef119e4a48966766094aeb0b94 (table rebalancer_dynamic_table_new_name [id=3805d326c7e843e1b557de84dc07d6ce]) successfully deleted
I20240501 14:09:49.490664 6241 ts_tablet_manager.cc:1888] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:49.490763 6241 log.cc:1196] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/4cd842ef119e4a48966766094aeb0b94
I20240501 14:09:49.491107 6241 ts_tablet_manager.cc:1909] T 4cd842ef119e4a48966766094aeb0b94 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:49.491765 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 4cd842ef119e4a48966766094aeb0b94 (table rebalancer_dynamic_table_new_name [id=3805d326c7e843e1b557de84dc07d6ce]) successfully deleted
I20240501 14:09:49.504998 30013 consensus_queue.cc:237] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:49.506191 32553 raft_consensus.cc:1261] T deee2ab875274d4eaeab84e698241ca8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:49.506206 30406 raft_consensus.cc:1261] T deee2ab875274d4eaeab84e698241ca8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:49.506531 6075 consensus_queue.cc:1035] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:49.506848 5730 consensus_queue.cc:1035] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:49.508613 6261 raft_consensus.cc:2932] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:49.508992 30406 raft_consensus.cc:2932] T deee2ab875274d4eaeab84e698241ca8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:49.509002 32553 raft_consensus.cc:2932] T deee2ab875274d4eaeab84e698241ca8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:49.512887 29868 catalog_manager.cc:5484] T deee2ab875274d4eaeab84e698241ca8 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:49.516108 30255 tablet_service.cc:1514] Processing DeleteTablet for tablet deee2ab875274d4eaeab84e698241ca8 with delete_type TABLET_DATA_TOMBSTONED (TS 1d0bb69611af4876a2b3b33c4766e13a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:49.517153 6262 tablet_replica.cc:330] T deee2ab875274d4eaeab84e698241ca8 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:49.517500 6262 raft_consensus.cc:2229] T deee2ab875274d4eaeab84e698241ca8 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:49.517679 6262 raft_consensus.cc:2258] T deee2ab875274d4eaeab84e698241ca8 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:49.518332 6262 ts_tablet_manager.cc:1875] T deee2ab875274d4eaeab84e698241ca8 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:49.521248 6262 ts_tablet_manager.cc:1888] T deee2ab875274d4eaeab84e698241ca8 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:49.521363 6262 log.cc:1196] T deee2ab875274d4eaeab84e698241ca8 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/deee2ab875274d4eaeab84e698241ca8
I20240501 14:09:49.521826 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet deee2ab875274d4eaeab84e698241ca8 (table rebalance_test_table_15 [id=df0504ee71e643e3a5ec9a20e1037981]) successfully deleted
I20240501 14:09:49.583143 29873 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:49.588275 32533 tablet_service.cc:1467] Processing CreateTablet for tablet 7d2b39cde4c0498e93076aa8b5dca44b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1dd1e9f3d05b48d8af630d6a413616db]), partition=
I20240501 14:09:49.588515 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d2b39cde4c0498e93076aa8b5dca44b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.588483 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 7d2b39cde4c0498e93076aa8b5dca44b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1dd1e9f3d05b48d8af630d6a413616db]), partition=
I20240501 14:09:49.588734 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d2b39cde4c0498e93076aa8b5dca44b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.591784 329 tablet_service.cc:1467] Processing CreateTablet for tablet 7d2b39cde4c0498e93076aa8b5dca44b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1dd1e9f3d05b48d8af630d6a413616db]), partition=
I20240501 14:09:49.592133 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d2b39cde4c0498e93076aa8b5dca44b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.597164 6159 tablet_bootstrap.cc:492] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:49.598151 6159 tablet_bootstrap.cc:654] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.598598 6155 tablet_bootstrap.cc:492] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:49.599360 6159 tablet_bootstrap.cc:492] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:49.599443 6159 ts_tablet_manager.cc:1367] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:49.599473 6155 tablet_bootstrap.cc:654] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.599885 6159 raft_consensus.cc:348] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:49.600009 6159 raft_consensus.cc:374] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.600039 6159 raft_consensus.cc:729] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.600132 6159 consensus_queue.cc:260] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:49.600534 6304 tablet_bootstrap.cc:492] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:49.602787 6304 tablet_bootstrap.cc:654] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.610580 6155 tablet_bootstrap.cc:492] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:49.610678 6155 ts_tablet_manager.cc:1367] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 14:09:49.611155 6155 raft_consensus.cc:348] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:49.611284 6155 raft_consensus.cc:374] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.611322 6155 raft_consensus.cc:729] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.611433 6155 consensus_queue.cc:260] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:49.611858 6155 ts_tablet_manager.cc:1398] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:49.613569 6159 ts_tablet_manager.cc:1398] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240501 14:09:49.616442 6304 tablet_bootstrap.cc:492] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:49.616616 6304 ts_tablet_manager.cc:1367] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:09:49.617151 6304 raft_consensus.cc:348] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:49.617311 6304 raft_consensus.cc:374] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.617388 6304 raft_consensus.cc:729] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.617619 6304 consensus_queue.cc:260] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:49.618782 6304 ts_tablet_manager.cc:1398] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:49.633510 30388 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:49.643252 30255 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:49.646109 329 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:49.647014 29993 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:49.652343 32664 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:49.655797 32533 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:49.673244 30126 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:49.675276 32401 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:49.726488 32415 consensus_queue.cc:237] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:49.727588 30145 raft_consensus.cc:1261] T 2c5c502325fb475f8f050ccad34feb2a P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:49.727638 32553 raft_consensus.cc:1261] T 2c5c502325fb475f8f050ccad34feb2a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:49.728000 6003 consensus_queue.cc:1035] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:49.728305 6133 consensus_queue.cc:1035] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:49.730001 6003 raft_consensus.cc:2932] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:49.730381 30145 raft_consensus.cc:2932] T 2c5c502325fb475f8f050ccad34feb2a P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:49.731634 32553 raft_consensus.cc:2932] T 2c5c502325fb475f8f050ccad34feb2a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:49.735692 29870 catalog_manager.cc:5484] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:49.739697 32664 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c5c502325fb475f8f050ccad34feb2a with delete_type TABLET_DATA_TOMBSTONED (TS db935501e9104de5b15c8780681d6c38 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:49.739920 6242 tablet_replica.cc:330] T 2c5c502325fb475f8f050ccad34feb2a P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:49.740074 6242 raft_consensus.cc:2229] T 2c5c502325fb475f8f050ccad34feb2a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:49.740198 6242 raft_consensus.cc:2258] T 2c5c502325fb475f8f050ccad34feb2a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:49.740687 6242 ts_tablet_manager.cc:1875] T 2c5c502325fb475f8f050ccad34feb2a P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:49.742430 6242 ts_tablet_manager.cc:1888] T 2c5c502325fb475f8f050ccad34feb2a P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:09:49.742499 6242 log.cc:1196] T 2c5c502325fb475f8f050ccad34feb2a P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/2c5c502325fb475f8f050ccad34feb2a
I20240501 14:09:49.742915 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet 2c5c502325fb475f8f050ccad34feb2a (table rebalancer_extra_table_2 [id=e53010e4736c42a899edac137f9182cb]) successfully deleted
I20240501 14:09:49.743422 4830 consensus_queue.cc:1035] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:49.747468 6179 consensus_queue.cc:1035] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:49.764079 6004 raft_consensus.cc:482] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:49.764184 6004 raft_consensus.cc:504] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:49.764660 6004 leader_election.cc:288] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:49.764928 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d2b39cde4c0498e93076aa8b5dca44b" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:49.765074 348 raft_consensus.cc:2454] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:49.765059 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d2b39cde4c0498e93076aa8b5dca44b" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:49.765172 30014 raft_consensus.cc:2454] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:49.765368 32489 leader_election.cc:302] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:49.765506 6004 raft_consensus.cc:2781] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:49.765573 6004 raft_consensus.cc:482] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:49.765609 6004 raft_consensus.cc:3037] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.766600 6004 raft_consensus.cc:504] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:49.766897 6004 leader_election.cc:288] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:49.767247 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d2b39cde4c0498e93076aa8b5dca44b" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:49.767297 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d2b39cde4c0498e93076aa8b5dca44b" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:49.767355 348 raft_consensus.cc:3037] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.767410 30014 raft_consensus.cc:3037] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.768319 30014 raft_consensus.cc:2454] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:49.768318 348 raft_consensus.cc:2454] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:49.768615 32489 leader_election.cc:302] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:49.768828 6004 raft_consensus.cc:2781] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:49.768932 6004 raft_consensus.cc:686] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:09:49.769070 6004 consensus_queue.cc:237] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:49.771081 29864 catalog_manager.cc:5484] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:49.805181 5812 consensus_queue.cc:1035] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:49.816766 5812 consensus_queue.cc:1035] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:49.853868 6028 consensus_queue.cc:1035] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:49.858263 30255 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:49.862883 32533 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:49.863281 30388 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:49.865617 6185 consensus_queue.cc:1035] T 745976e20bdc47268fad141ceccd1a81 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:49.875546 29993 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:49.875788 329 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:49.880102 32401 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:49.882016 29873 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:49.888411 32664 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:49.890589 30014 raft_consensus.cc:1261] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:49.890957 5552 consensus_queue.cc:1035] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:49.892130 30126 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:49.892436 348 raft_consensus.cc:1261] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:49.892853 5552 consensus_queue.cc:1035] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:49.893792 29873 catalog_manager.cc:5576] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387) does not have the latest schema for tablet 7d2b39cde4c0498e93076aa8b5dca44b (table rebalancer_dynamic_table [id=1dd1e9f3d05b48d8af630d6a413616db]). Expected version 1 got 0
I20240501 14:09:49.895360 29873 catalog_manager.cc:5576] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937) does not have the latest schema for tablet 7d2b39cde4c0498e93076aa8b5dca44b (table rebalancer_dynamic_table [id=1dd1e9f3d05b48d8af630d6a413616db]). Expected version 1 got 0
I20240501 14:09:49.903527 6405 tablet.cc:1722] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:49.903910 6405 tablet.cc:1620] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e: MemRowSet was empty: no flush needed.
I20240501 14:09:49.903900 29873 catalog_manager.cc:5576] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387) does not have the latest schema for tablet 7d2b39cde4c0498e93076aa8b5dca44b (table rebalancer_dynamic_table [id=1dd1e9f3d05b48d8af630d6a413616db]). Expected version 1 got 0
I20240501 14:09:49.904158 6404 tablet.cc:1722] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:49.906266 6404 tablet.cc:1620] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:49.911059 29865 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=1dd1e9f3d05b48d8af630d6a413616db] alter complete (version 1)
I20240501 14:09:49.930871 29873 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:49.933574 29865 catalog_manager.cc:5576] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937) does not have the latest schema for tablet 7d2b39cde4c0498e93076aa8b5dca44b (table rebalancer_dynamic_table [id=1dd1e9f3d05b48d8af630d6a413616db]). Expected version 1 got 0
I20240501 14:09:49.935091 6407 tablet.cc:1722] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:49.940256 6407 tablet.cc:1620] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:49.940759 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 270f2887aa3d43449695c61e2c2ae1d6 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=67fbf296b4324bdb855fc05097cf2d0f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.941198 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270f2887aa3d43449695c61e2c2ae1d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.941134 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 06db74bb439845d4bf043c5047c65da4 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=67fbf296b4324bdb855fc05097cf2d0f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.941663 29995 tablet_service.cc:1467] Processing CreateTablet for tablet e7ae4cac38e0454d8b6e019208553dac (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=67fbf296b4324bdb855fc05097cf2d0f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.941892 29995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7ae4cac38e0454d8b6e019208553dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.943850 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 270f2887aa3d43449695c61e2c2ae1d6 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=67fbf296b4324bdb855fc05097cf2d0f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.944061 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270f2887aa3d43449695c61e2c2ae1d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.944139 6304 tablet_bootstrap.cc:492] T e7ae4cac38e0454d8b6e019208553dac P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:49.945115 32400 tablet_service.cc:1467] Processing CreateTablet for tablet e7ae4cac38e0454d8b6e019208553dac (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=67fbf296b4324bdb855fc05097cf2d0f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.945201 6304 tablet_bootstrap.cc:654] T e7ae4cac38e0454d8b6e019208553dac P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.945417 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7ae4cac38e0454d8b6e019208553dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.945914 32401 tablet_service.cc:1467] Processing CreateTablet for tablet 270f2887aa3d43449695c61e2c2ae1d6 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=67fbf296b4324bdb855fc05097cf2d0f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.945956 30255 tablet_service.cc:1467] Processing CreateTablet for tablet e7ae4cac38e0454d8b6e019208553dac (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=67fbf296b4324bdb855fc05097cf2d0f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.946121 32401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270f2887aa3d43449695c61e2c2ae1d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.946174 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7ae4cac38e0454d8b6e019208553dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.947633 6415 tablet_bootstrap.cc:492] T e7ae4cac38e0454d8b6e019208553dac P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:49.948779 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06db74bb439845d4bf043c5047c65da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.949304 6304 tablet_bootstrap.cc:492] T e7ae4cac38e0454d8b6e019208553dac P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:49.949374 6304 ts_tablet_manager.cc:1367] T e7ae4cac38e0454d8b6e019208553dac P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:49.949838 6304 raft_consensus.cc:348] T e7ae4cac38e0454d8b6e019208553dac P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:49.949937 6304 raft_consensus.cc:374] T e7ae4cac38e0454d8b6e019208553dac P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.949967 6304 raft_consensus.cc:729] T e7ae4cac38e0454d8b6e019208553dac P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.950033 6415 tablet_bootstrap.cc:654] T e7ae4cac38e0454d8b6e019208553dac P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.950110 6304 consensus_queue.cc:260] T e7ae4cac38e0454d8b6e019208553dac P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:49.950227 329 tablet_service.cc:1467] Processing CreateTablet for tablet 06db74bb439845d4bf043c5047c65da4 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=67fbf296b4324bdb855fc05097cf2d0f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.950452 6304 ts_tablet_manager.cc:1398] T e7ae4cac38e0454d8b6e019208553dac P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:49.950491 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06db74bb439845d4bf043c5047c65da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.952343 6413 tablet_bootstrap.cc:492] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:49.952960 30126 tablet_service.cc:1467] Processing CreateTablet for tablet 06db74bb439845d4bf043c5047c65da4 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=67fbf296b4324bdb855fc05097cf2d0f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.953248 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06db74bb439845d4bf043c5047c65da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.952169 6304 tablet_bootstrap.cc:492] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:49.955979 6304 tablet_bootstrap.cc:654] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.956193 6413 tablet_bootstrap.cc:654] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.957518 6415 tablet_bootstrap.cc:492] T e7ae4cac38e0454d8b6e019208553dac P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:49.957638 6415 ts_tablet_manager.cc:1367] T e7ae4cac38e0454d8b6e019208553dac P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:49.958148 6415 raft_consensus.cc:348] T e7ae4cac38e0454d8b6e019208553dac P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:49.958344 6415 raft_consensus.cc:374] T e7ae4cac38e0454d8b6e019208553dac P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.958395 6155 tablet_bootstrap.cc:492] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:49.958438 6415 raft_consensus.cc:729] T e7ae4cac38e0454d8b6e019208553dac P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.958597 6415 consensus_queue.cc:260] T e7ae4cac38e0454d8b6e019208553dac P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:49.959290 6155 tablet_bootstrap.cc:654] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.959380 6415 ts_tablet_manager.cc:1398] T e7ae4cac38e0454d8b6e019208553dac P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:49.960203 6415 tablet_bootstrap.cc:492] T 270f2887aa3d43449695c61e2c2ae1d6 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:49.961161 6413 tablet_bootstrap.cc:492] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:49.961252 6413 ts_tablet_manager.cc:1367] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240501 14:09:49.961714 6413 raft_consensus.cc:348] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:49.961813 6413 raft_consensus.cc:374] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.961846 6413 raft_consensus.cc:729] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.962143 6416 tablet_bootstrap.cc:492] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:49.962486 6415 tablet_bootstrap.cc:654] T 270f2887aa3d43449695c61e2c2ae1d6 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.962810 6155 tablet_bootstrap.cc:492] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:49.962884 6155 ts_tablet_manager.cc:1367] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:49.963340 6155 raft_consensus.cc:348] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:49.963428 6155 raft_consensus.cc:374] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.963452 6155 raft_consensus.cc:729] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.963541 6155 consensus_queue.cc:260] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:49.963752 6155 ts_tablet_manager.cc:1398] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:49.964321 6416 tablet_bootstrap.cc:654] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.965023 6423 tablet_bootstrap.cc:492] T 06db74bb439845d4bf043c5047c65da4 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:49.963024 6413 consensus_queue.cc:260] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:49.966044 6413 ts_tablet_manager.cc:1398] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:09:49.966255 6413 tablet_bootstrap.cc:492] T 06db74bb439845d4bf043c5047c65da4 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:49.967466 6423 tablet_bootstrap.cc:654] T 06db74bb439845d4bf043c5047c65da4 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.968555 6413 tablet_bootstrap.cc:654] T 06db74bb439845d4bf043c5047c65da4 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.968672 6304 tablet_bootstrap.cc:492] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:49.968770 6304 ts_tablet_manager.cc:1367] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 14:09:49.968956 6416 tablet_bootstrap.cc:492] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:49.969053 6416 ts_tablet_manager.cc:1367] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240501 14:09:49.969592 6416 raft_consensus.cc:348] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:49.969967 6416 raft_consensus.cc:374] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.970026 6416 raft_consensus.cc:729] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.970175 6416 consensus_queue.cc:260] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:49.970527 6416 ts_tablet_manager.cc:1398] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:49.970871 6304 raft_consensus.cc:348] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:49.971002 6304 raft_consensus.cc:374] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.971045 6304 raft_consensus.cc:729] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.971371 6304 consensus_queue.cc:260] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:49.971910 6415 tablet_bootstrap.cc:492] T 270f2887aa3d43449695c61e2c2ae1d6 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:49.972050 6415 ts_tablet_manager.cc:1367] T 270f2887aa3d43449695c61e2c2ae1d6 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:49.972573 6415 raft_consensus.cc:348] T 270f2887aa3d43449695c61e2c2ae1d6 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:49.972702 6415 raft_consensus.cc:374] T 270f2887aa3d43449695c61e2c2ae1d6 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.972752 6415 raft_consensus.cc:729] T 270f2887aa3d43449695c61e2c2ae1d6 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.972892 6415 consensus_queue.cc:260] T 270f2887aa3d43449695c61e2c2ae1d6 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:49.973835 6415 ts_tablet_manager.cc:1398] T 270f2887aa3d43449695c61e2c2ae1d6 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:49.974450 6413 tablet_bootstrap.cc:492] T 06db74bb439845d4bf043c5047c65da4 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:49.974539 6413 ts_tablet_manager.cc:1367] T 06db74bb439845d4bf043c5047c65da4 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240501 14:09:49.974877 6304 ts_tablet_manager.cc:1398] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:49.974911 6413 raft_consensus.cc:348] T 06db74bb439845d4bf043c5047c65da4 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:49.975034 6413 raft_consensus.cc:374] T 06db74bb439845d4bf043c5047c65da4 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.975077 6413 raft_consensus.cc:729] T 06db74bb439845d4bf043c5047c65da4 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.975178 6413 consensus_queue.cc:260] T 06db74bb439845d4bf043c5047c65da4 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:49.975984 6413 ts_tablet_manager.cc:1398] T 06db74bb439845d4bf043c5047c65da4 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:49.977027 6423 tablet_bootstrap.cc:492] T 06db74bb439845d4bf043c5047c65da4 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:49.977141 6423 ts_tablet_manager.cc:1367] T 06db74bb439845d4bf043c5047c65da4 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:49.978612 6423 raft_consensus.cc:348] T 06db74bb439845d4bf043c5047c65da4 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:49.978758 6423 raft_consensus.cc:374] T 06db74bb439845d4bf043c5047c65da4 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.978811 6423 raft_consensus.cc:729] T 06db74bb439845d4bf043c5047c65da4 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.979851 6423 consensus_queue.cc:260] T 06db74bb439845d4bf043c5047c65da4 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:49.981290 6423 ts_tablet_manager.cc:1398] T 06db74bb439845d4bf043c5047c65da4 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:49.993161 30014 consensus_queue.cc:237] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:49.994385 32415 raft_consensus.cc:1261] T 15c98173a71d4f73b8f6af24ad01d7c4 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:49.994439 30145 raft_consensus.cc:1261] T 15c98173a71d4f73b8f6af24ad01d7c4 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:49.994762 5730 consensus_queue.cc:1035] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:49.995729 5730 consensus_queue.cc:1035] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:49.996604 5730 raft_consensus.cc:2932] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:49.998522 30145 raft_consensus.cc:2932] T 15c98173a71d4f73b8f6af24ad01d7c4 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:49.999765 29869 catalog_manager.cc:5484] T 15c98173a71d4f73b8f6af24ad01d7c4 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:50.003647 30255 tablet_service.cc:1514] Processing DeleteTablet for tablet 15c98173a71d4f73b8f6af24ad01d7c4 with delete_type TABLET_DATA_TOMBSTONED (TS 1d0bb69611af4876a2b3b33c4766e13a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:50.003872 6262 tablet_replica.cc:330] T 15c98173a71d4f73b8f6af24ad01d7c4 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:50.004078 6262 raft_consensus.cc:2229] T 15c98173a71d4f73b8f6af24ad01d7c4 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:50.004273 6262 raft_consensus.cc:2258] T 15c98173a71d4f73b8f6af24ad01d7c4 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:50.004856 6262 ts_tablet_manager.cc:1875] T 15c98173a71d4f73b8f6af24ad01d7c4 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:50.006919 6262 ts_tablet_manager.cc:1888] T 15c98173a71d4f73b8f6af24ad01d7c4 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:50.006999 6262 log.cc:1196] T 15c98173a71d4f73b8f6af24ad01d7c4 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/15c98173a71d4f73b8f6af24ad01d7c4
I20240501 14:09:50.007925 32415 raft_consensus.cc:2932] T 15c98173a71d4f73b8f6af24ad01d7c4 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:50.008563 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet 15c98173a71d4f73b8f6af24ad01d7c4 (table rebalance_test_table_15 [id=df0504ee71e643e3a5ec9a20e1037981]) successfully deleted
I20240501 14:09:50.011346 6179 raft_consensus.cc:482] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:50.011452 6179 raft_consensus.cc:504] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:50.011871 6179 leader_election.cc:288] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:50.012478 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270f2887aa3d43449695c61e2c2ae1d6" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:50.012603 32415 raft_consensus.cc:2454] T 270f2887aa3d43449695c61e2c2ae1d6 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:50.013898 32620 leader_election.cc:302] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:50.014097 6179 raft_consensus.cc:2781] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:50.014137 6179 raft_consensus.cc:482] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:50.014169 6179 raft_consensus.cc:3037] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.015297 6179 raft_consensus.cc:504] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:50.015643 6179 leader_election.cc:288] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:50.016996 29869 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:50.019748 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270f2887aa3d43449695c61e2c2ae1d6" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:50.019845 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270f2887aa3d43449695c61e2c2ae1d6" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:50.019865 30013 raft_consensus.cc:3037] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.020365 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270f2887aa3d43449695c61e2c2ae1d6" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:50.020457 32415 raft_consensus.cc:3037] T 270f2887aa3d43449695c61e2c2ae1d6 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.021142 30013 raft_consensus.cc:2454] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:50.021417 32415 raft_consensus.cc:2454] T 270f2887aa3d43449695c61e2c2ae1d6 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:50.021888 32620 leader_election.cc:302] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:50.022058 6179 raft_consensus.cc:2781] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:50.022128 6179 raft_consensus.cc:686] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Becoming Leader. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
I20240501 14:09:50.022243 6179 consensus_queue.cc:237] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:50.023763 6404 tablet.cc:1722] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:50.024080 6404 tablet.cc:1620] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:50.024225 6407 tablet.cc:1722] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:50.024530 6407 tablet.cc:1620] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:50.025724 29869 catalog_manager.cc:5484] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db935501e9104de5b15c8780681d6c38 (127.22.252.135). New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:50.026188 6405 tablet.cc:1722] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:50.026463 6405 tablet.cc:1620] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e: MemRowSet was empty: no flush needed.
I20240501 14:09:50.031751 29873 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=1dd1e9f3d05b48d8af630d6a413616db] alter complete (version 2)
I20240501 14:09:50.137208 29868 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:50.142616 6404 tablet.cc:1722] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:50.144995 6405 tablet.cc:1722] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:50.145097 6407 tablet.cc:1722] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:50.148864 6433 raft_consensus.cc:482] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:50.148989 6433 raft_consensus.cc:504] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:50.149412 6433 leader_election.cc:288] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:50.149832 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ae4cac38e0454d8b6e019208553dac" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:50.149983 32415 raft_consensus.cc:2454] T e7ae4cac38e0454d8b6e019208553dac P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:50.149966 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ae4cac38e0454d8b6e019208553dac" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:50.150080 30013 raft_consensus.cc:2454] T e7ae4cac38e0454d8b6e019208553dac P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:50.150336 30212 leader_election.cc:302] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:50.150519 6433 raft_consensus.cc:2781] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:50.150590 6433 raft_consensus.cc:482] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:50.150643 6433 raft_consensus.cc:3037] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.151504 6433 raft_consensus.cc:504] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:50.151798 6433 leader_election.cc:288] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:50.152108 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ae4cac38e0454d8b6e019208553dac" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:50.152138 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ae4cac38e0454d8b6e019208553dac" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:50.152213 30013 raft_consensus.cc:3037] T e7ae4cac38e0454d8b6e019208553dac P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.152259 32415 raft_consensus.cc:3037] T e7ae4cac38e0454d8b6e019208553dac P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.153014 29868 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=1dd1e9f3d05b48d8af630d6a413616db] alter complete (version 3)
I20240501 14:09:50.153255 30013 raft_consensus.cc:2454] T e7ae4cac38e0454d8b6e019208553dac P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:50.153263 32415 raft_consensus.cc:2454] T e7ae4cac38e0454d8b6e019208553dac P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:50.153856 30212 leader_election.cc:302] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 93f2518fbadf4f9991cda93afacb363e; no voters:
I20240501 14:09:50.154158 6433 raft_consensus.cc:2781] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:50.154255 6433 raft_consensus.cc:686] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:50.154394 6433 consensus_queue.cc:237] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:50.156613 29864 catalog_manager.cc:5484] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:50.254832 29873 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:50.255045 29873 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:50.257040 29873 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 7d2b39cde4c0498e93076aa8b5dca44b
I20240501 14:09:50.257637 32533 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d2b39cde4c0498e93076aa8b5dca44b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:50 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:50.257637 329 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d2b39cde4c0498e93076aa8b5dca44b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:50 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:50.257905 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d2b39cde4c0498e93076aa8b5dca44b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:50 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:50.258023 6488 tablet_replica.cc:330] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:50.258198 6488 raft_consensus.cc:2229] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:50.258368 6490 tablet_replica.cc:330] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:50.258368 6488 raft_consensus.cc:2258] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:50.258508 6490 raft_consensus.cc:2229] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:50.258575 6489 tablet_replica.cc:330] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:50.258666 6490 raft_consensus.cc:2258] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:50.258749 6489 raft_consensus.cc:2229] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:50.258967 6489 raft_consensus.cc:2258] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:50.259248 6488 ts_tablet_manager.cc:1875] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:50.259248 6490 ts_tablet_manager.cc:1875] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:50.259650 6489 ts_tablet_manager.cc:1875] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:50.261448 6489 ts_tablet_manager.cc:1888] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:50.261514 6489 log.cc:1196] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/7d2b39cde4c0498e93076aa8b5dca44b
I20240501 14:09:50.261507 6490 ts_tablet_manager.cc:1888] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:50.261572 6490 log.cc:1196] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/7d2b39cde4c0498e93076aa8b5dca44b
I20240501 14:09:50.261801 6489 ts_tablet_manager.cc:1909] T 7d2b39cde4c0498e93076aa8b5dca44b P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:50.261828 6488 ts_tablet_manager.cc:1888] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:50.261927 6490 ts_tablet_manager.cc:1909] T 7d2b39cde4c0498e93076aa8b5dca44b P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:50.261952 6488 log.cc:1196] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/7d2b39cde4c0498e93076aa8b5dca44b
I20240501 14:09:50.262255 6488 ts_tablet_manager.cc:1909] T 7d2b39cde4c0498e93076aa8b5dca44b P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:09:50.262537 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet 7d2b39cde4c0498e93076aa8b5dca44b (table rebalancer_dynamic_table_new_name [id=1dd1e9f3d05b48d8af630d6a413616db]) successfully deleted
I20240501 14:09:50.262805 6424 raft_consensus.cc:482] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:50.262893 6424 raft_consensus.cc:504] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:50.263348 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 7d2b39cde4c0498e93076aa8b5dca44b (table rebalancer_dynamic_table_new_name [id=1dd1e9f3d05b48d8af630d6a413616db]) successfully deleted
I20240501 14:09:50.263360 6424 leader_election.cc:288] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:50.263891 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06db74bb439845d4bf043c5047c65da4" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:50.263911 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06db74bb439845d4bf043c5047c65da4" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:50.264029 32683 raft_consensus.cc:2454] T 06db74bb439845d4bf043c5047c65da4 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:50.264048 30145 raft_consensus.cc:2454] T 06db74bb439845d4bf043c5047c65da4 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:50.264068 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 7d2b39cde4c0498e93076aa8b5dca44b (table rebalancer_dynamic_table_new_name [id=1dd1e9f3d05b48d8af630d6a413616db]) successfully deleted
I20240501 14:09:50.264331 32750 leader_election.cc:302] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:50.264483 6424 raft_consensus.cc:2781] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:50.264567 6424 raft_consensus.cc:482] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:50.264611 6424 raft_consensus.cc:3037] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.265347 6424 raft_consensus.cc:504] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:50.265636 6424 leader_election.cc:288] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:50.265863 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06db74bb439845d4bf043c5047c65da4" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:50.265972 30145 raft_consensus.cc:3037] T 06db74bb439845d4bf043c5047c65da4 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.265955 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06db74bb439845d4bf043c5047c65da4" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:50.266052 32683 raft_consensus.cc:3037] T 06db74bb439845d4bf043c5047c65da4 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.266901 30145 raft_consensus.cc:2454] T 06db74bb439845d4bf043c5047c65da4 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:50.266942 32683 raft_consensus.cc:2454] T 06db74bb439845d4bf043c5047c65da4 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:50.267199 32753 leader_election.cc:302] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:50.267357 6424 raft_consensus.cc:2781] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:50.267441 6424 raft_consensus.cc:686] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:50.267562 6424 consensus_queue.cc:237] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:50.269376 29873 catalog_manager.cc:5484] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:50.297000 32533 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:50.300251 30388 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:50.302340 30126 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:50.303897 32665 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:50.312968 29993 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:50.316455 32401 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:50.317471 329 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:50.340874 30255 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:50.359270 29873 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:50.364988 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 60096eb27894484190bd5a3a23f03e68 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2ae45ec6a8f0431181d15f77d852cb3c]), partition=
I20240501 14:09:50.365000 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 60096eb27894484190bd5a3a23f03e68 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2ae45ec6a8f0431181d15f77d852cb3c]), partition=
I20240501 14:09:50.365247 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60096eb27894484190bd5a3a23f03e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.365247 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60096eb27894484190bd5a3a23f03e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.365219 32533 tablet_service.cc:1467] Processing CreateTablet for tablet 60096eb27894484190bd5a3a23f03e68 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2ae45ec6a8f0431181d15f77d852cb3c]), partition=
I20240501 14:09:50.365455 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60096eb27894484190bd5a3a23f03e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.367442 6413 tablet_bootstrap.cc:492] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:50.367849 6504 tablet_bootstrap.cc:492] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:50.368971 6413 tablet_bootstrap.cc:654] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.369124 6504 tablet_bootstrap.cc:654] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.370227 6504 tablet_bootstrap.cc:492] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:50.370383 6504 ts_tablet_manager.cc:1367] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:50.370846 6504 raft_consensus.cc:348] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:50.370973 6413 tablet_bootstrap.cc:492] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:50.371043 6413 ts_tablet_manager.cc:1367] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:09:50.371122 6504 raft_consensus.cc:374] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.371181 6504 raft_consensus.cc:729] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.371328 6504 consensus_queue.cc:260] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:50.371450 6413 raft_consensus.cc:348] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:50.371544 6413 raft_consensus.cc:374] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.371573 6413 raft_consensus.cc:729] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.371677 6504 ts_tablet_manager.cc:1398] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:50.371667 6413 consensus_queue.cc:260] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:50.371897 6413 ts_tablet_manager.cc:1398] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:50.372437 6503 tablet_bootstrap.cc:492] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:50.373389 6503 tablet_bootstrap.cc:654] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.374423 6503 tablet_bootstrap.cc:492] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:50.374506 6503 ts_tablet_manager.cc:1367] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:50.374939 6503 raft_consensus.cc:348] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:50.375030 6503 raft_consensus.cc:374] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.375056 6503 raft_consensus.cc:729] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.375149 6503 consensus_queue.cc:260] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:50.375430 6503 ts_tablet_manager.cc:1398] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:50.396971 32415 consensus_queue.cc:237] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:50.398056 32553 raft_consensus.cc:1261] T 7a6a29261186429cbc05b1819999f986 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:50.398068 30406 raft_consensus.cc:1261] T 7a6a29261186429cbc05b1819999f986 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:50.398450 5812 consensus_queue.cc:1035] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:50.398607 6133 consensus_queue.cc:1035] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:50.400350 6526 raft_consensus.cc:2932] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:50.400687 32553 raft_consensus.cc:2932] T 7a6a29261186429cbc05b1819999f986 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:50.400748 30406 raft_consensus.cc:2932] T 7a6a29261186429cbc05b1819999f986 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:50.404522 29868 catalog_manager.cc:5484] T 7a6a29261186429cbc05b1819999f986 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:50.406898 329 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a6a29261186429cbc05b1819999f986 with delete_type TABLET_DATA_TOMBSTONED (TS 9ecd907cf5634ac98f3f54f3dbec6849 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:50.407116 6488 tablet_replica.cc:330] T 7a6a29261186429cbc05b1819999f986 P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:50.407275 6488 raft_consensus.cc:2229] T 7a6a29261186429cbc05b1819999f986 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:50.407390 6488 raft_consensus.cc:2258] T 7a6a29261186429cbc05b1819999f986 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:50.407814 6488 ts_tablet_manager.cc:1875] T 7a6a29261186429cbc05b1819999f986 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:50.409559 6456 raft_consensus.cc:482] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:50.409646 6456 raft_consensus.cc:504] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:50.410017 6488 ts_tablet_manager.cc:1888] T 7a6a29261186429cbc05b1819999f986 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:09:50.410072 6488 log.cc:1196] T 7a6a29261186429cbc05b1819999f986 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/7a6a29261186429cbc05b1819999f986
I20240501 14:09:50.410167 6456 leader_election.cc:288] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:50.410521 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60096eb27894484190bd5a3a23f03e68" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:50.410648 32683 raft_consensus.cc:2454] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:50.410893 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 7a6a29261186429cbc05b1819999f986 (table rebalancer_extra_table_4 [id=ec60e0d43df9403baaddddba692166ca]) successfully deleted
I20240501 14:09:50.411273 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60096eb27894484190bd5a3a23f03e68" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:50.411402 30406 raft_consensus.cc:2454] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:50.411520 32487 leader_election.cc:302] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:50.411790 5552 raft_consensus.cc:2781] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:50.411855 5552 raft_consensus.cc:482] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:50.411893 5552 raft_consensus.cc:3037] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.413317 5552 raft_consensus.cc:504] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:50.413642 5552 leader_election.cc:288] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:50.417021 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60096eb27894484190bd5a3a23f03e68" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:50.417063 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60096eb27894484190bd5a3a23f03e68" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:50.417141 30406 raft_consensus.cc:3037] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.417178 32683 raft_consensus.cc:3037] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.418183 30406 raft_consensus.cc:2454] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:50.418238 32683 raft_consensus.cc:2454] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:50.418493 32487 leader_election.cc:302] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:50.418689 6456 raft_consensus.cc:2781] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:50.418769 6456 raft_consensus.cc:686] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:09:50.418910 6456 consensus_queue.cc:237] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:50.421070 29864 catalog_manager.cc:5484] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:50.507365 29993 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:50.512691 32533 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:50.520519 32401 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:50.520961 30255 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:50.523253 30388 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:50.523626 329 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:50.526535 29864 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:50.533658 32665 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:50.534206 30126 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:50.548812 32683 raft_consensus.cc:1261] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:50.549275 30406 raft_consensus.cc:1261] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:50.550130 5552 consensus_queue.cc:1035] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:50.550393 5552 consensus_queue.cc:1035] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:50.554764 29864 catalog_manager.cc:5576] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357) does not have the latest schema for tablet 60096eb27894484190bd5a3a23f03e68 (table rebalancer_dynamic_table [id=2ae45ec6a8f0431181d15f77d852cb3c]). Expected version 1 got 0
I20240501 14:09:50.556638 6404 tablet.cc:1722] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:50.556957 6404 tablet.cc:1620] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:50.558033 29864 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet 60096eb27894484190bd5a3a23f03e68 (table rebalancer_dynamic_table [id=2ae45ec6a8f0431181d15f77d852cb3c]). Expected version 1 got 0
I20240501 14:09:50.562368 29864 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet 60096eb27894484190bd5a3a23f03e68 (table rebalancer_dynamic_table [id=2ae45ec6a8f0431181d15f77d852cb3c]). Expected version 1 got 0
I20240501 14:09:50.564525 29866 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=2ae45ec6a8f0431181d15f77d852cb3c] alter complete (version 1)
I20240501 14:09:50.567570 4830 consensus_queue.cc:1035] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:50.573390 6581 mvcc.cc:205] Tried to move back new op lower bound from 7022889330848038912 to 7022889330357112832. Current Snapshot: MvccSnapshot[applied={T|T < 7022889330848038912}]
I20240501 14:09:50.577879 4830 consensus_queue.cc:1035] T 270f2887aa3d43449695c61e2c2ae1d6 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:50.580480 29864 catalog_manager.cc:5576] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357) does not have the latest schema for tablet 60096eb27894484190bd5a3a23f03e68 (table rebalancer_dynamic_table [id=2ae45ec6a8f0431181d15f77d852cb3c]). Expected version 1 got 0
I20240501 14:09:50.581796 6583 mvcc.cc:205] Tried to move back new op lower bound from 7022889330848038912 to 7022889330357112832. Current Snapshot: MvccSnapshot[applied={T|T < 7022889330848038912}]
I20240501 14:09:50.582851 6587 tablet.cc:1722] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:50.583171 6587 tablet.cc:1620] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7: MemRowSet was empty: no flush needed.
I20240501 14:09:50.585032 6586 tablet.cc:1722] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:50.585283 6586 tablet.cc:1620] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:50.597396 6434 consensus_queue.cc:1035] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:50.602197 6433 consensus_queue.cc:1035] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:50.627190 32415 consensus_queue.cc:237] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:50.628680 349 raft_consensus.cc:1261] T b26b6d19699047afaee57d6f2c887664 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:50.629025 6526 consensus_queue.cc:1035] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:50.629274 32553 raft_consensus.cc:1261] T b26b6d19699047afaee57d6f2c887664 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:50.630069 6526 consensus_queue.cc:1035] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:50.631711 6133 raft_consensus.cc:2932] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:50.632300 349 raft_consensus.cc:2932] T b26b6d19699047afaee57d6f2c887664 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:50.635288 29866 catalog_manager.cc:5484] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:50.638504 32665 tablet_service.cc:1514] Processing DeleteTablet for tablet b26b6d19699047afaee57d6f2c887664 with delete_type TABLET_DATA_TOMBSTONED (TS db935501e9104de5b15c8780681d6c38 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:50.638440 32553 raft_consensus.cc:2932] T b26b6d19699047afaee57d6f2c887664 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:50.639865 6617 tablet_replica.cc:330] T b26b6d19699047afaee57d6f2c887664 P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:50.640533 6617 raft_consensus.cc:2229] T b26b6d19699047afaee57d6f2c887664 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:50.640712 6617 raft_consensus.cc:2258] T b26b6d19699047afaee57d6f2c887664 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:50.643718 6617 ts_tablet_manager.cc:1875] T b26b6d19699047afaee57d6f2c887664 P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:50.647321 6617 ts_tablet_manager.cc:1888] T b26b6d19699047afaee57d6f2c887664 P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:09:50.647518 6617 log.cc:1196] T b26b6d19699047afaee57d6f2c887664 P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/b26b6d19699047afaee57d6f2c887664
I20240501 14:09:50.648031 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet b26b6d19699047afaee57d6f2c887664 (table rebalancer_extra_table_10 [id=c8e44abe11554acda26e71b8a11e2e0e]) successfully deleted
I20240501 14:09:50.653329 5788 consensus_queue.cc:1035] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:50.656201 5788 consensus_queue.cc:1035] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:50.666730 29864 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:50.671643 6404 tablet.cc:1722] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:50.671981 6404 tablet.cc:1620] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:50.672909 6587 tablet.cc:1722] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:50.673205 6587 tablet.cc:1620] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7: MemRowSet was empty: no flush needed.
I20240501 14:09:50.675060 6586 tablet.cc:1722] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:50.675357 6586 tablet.cc:1620] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:50.677961 29866 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=2ae45ec6a8f0431181d15f77d852cb3c] alter complete (version 2)
I20240501 14:09:50.697930 30146 consensus_queue.cc:237] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:50.699682 30406 raft_consensus.cc:1261] T 529f938305214c3ba33f61e06d1acd77 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:50.699685 30277 raft_consensus.cc:1261] T 529f938305214c3ba33f61e06d1acd77 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:50.699817 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 529f938305214c3ba33f61e06d1acd77. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:50.700084 6406 consensus_queue.cc:1035] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:50.700475 6663 consensus_queue.cc:1035] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:50.702482 6406 raft_consensus.cc:2932] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:50.703799 30277 raft_consensus.cc:2932] T 529f938305214c3ba33f61e06d1acd77 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:50.705885 30406 raft_consensus.cc:2932] T 529f938305214c3ba33f61e06d1acd77 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:50.706607 29870 catalog_manager.cc:5484] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 reported cstate change: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:50.726181 30277 consensus_queue.cc:237] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:50.727365 30146 raft_consensus.cc:1261] T 19f203c753ef481dafc091ea2d60f56f P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:50.727492 30013 raft_consensus.cc:1261] T 19f203c753ef481dafc091ea2d60f56f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:50.727814 6433 consensus_queue.cc:1035] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:50.727969 6683 consensus_queue.cc:1035] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:50.728142 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f203c753ef481dafc091ea2d60f56f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:50.729636 6433 raft_consensus.cc:2932] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:50.729997 30146 raft_consensus.cc:2932] T 19f203c753ef481dafc091ea2d60f56f P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:50.732172 30013 raft_consensus.cc:2932] T 19f203c753ef481dafc091ea2d60f56f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:50.732749 29864 catalog_manager.cc:5484] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:50.751437 30277 consensus_queue.cc:237] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:50.752766 30013 raft_consensus.cc:1261] T f2bc06e76e2a41429d5625ebf99e5e5c P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:50.752871 30146 raft_consensus.cc:1261] T f2bc06e76e2a41429d5625ebf99e5e5c P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:50.753017 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2bc06e76e2a41429d5625ebf99e5e5c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:50.753154 6683 consensus_queue.cc:1035] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:50.753358 6686 consensus_queue.cc:1035] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:50.754921 6686 raft_consensus.cc:2932] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:50.755335 30146 raft_consensus.cc:2932] T f2bc06e76e2a41429d5625ebf99e5e5c P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:50.757232 30013 raft_consensus.cc:2932] T f2bc06e76e2a41429d5625ebf99e5e5c P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:50.759043 29870 catalog_manager.cc:5484] T f2bc06e76e2a41429d5625ebf99e5e5c P d35c16dcb506426993811bb2519d7db3 reported cstate change: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:50.781087 29864 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:50.786844 6404 tablet.cc:1722] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:50.787891 6587 tablet.cc:1722] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:50.789498 6586 tablet.cc:1722] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:50.792444 29864 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=2ae45ec6a8f0431181d15f77d852cb3c] alter complete (version 3)
I20240501 14:09:50.835563 29873 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:50.843221 32533 tablet_service.cc:1467] Processing CreateTablet for tablet 0cc8946505474737a610de35ddf0f2f6 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=44df1eb888434365abd5f82ea84af4e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:50.843529 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cc8946505474737a610de35ddf0f2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.843715 30126 tablet_service.cc:1467] Processing CreateTablet for tablet 7449ead7db7b4f7083aa4b7358fea0de (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=44df1eb888434365abd5f82ea84af4e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:50.844017 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7449ead7db7b4f7083aa4b7358fea0de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.844754 30255 tablet_service.cc:1467] Processing CreateTablet for tablet 7449ead7db7b4f7083aa4b7358fea0de (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=44df1eb888434365abd5f82ea84af4e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:50.845042 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7449ead7db7b4f7083aa4b7358fea0de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.845916 6504 tablet_bootstrap.cc:492] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:50.846638 32401 tablet_service.cc:1467] Processing CreateTablet for tablet ce9baa646d304268886d2cb76294c47c (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=44df1eb888434365abd5f82ea84af4e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:50.846801 6504 tablet_bootstrap.cc:654] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.846922 32401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce9baa646d304268886d2cb76294c47c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.847438 32664 tablet_service.cc:1467] Processing CreateTablet for tablet ce9baa646d304268886d2cb76294c47c (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=44df1eb888434365abd5f82ea84af4e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:50.847514 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 0cc8946505474737a610de35ddf0f2f6 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=44df1eb888434365abd5f82ea84af4e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:50.847687 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce9baa646d304268886d2cb76294c47c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.848038 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cc8946505474737a610de35ddf0f2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.848541 29993 tablet_service.cc:1467] Processing CreateTablet for tablet ce9baa646d304268886d2cb76294c47c (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=44df1eb888434365abd5f82ea84af4e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:50.848812 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce9baa646d304268886d2cb76294c47c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.850023 6413 tablet_bootstrap.cc:492] T ce9baa646d304268886d2cb76294c47c P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:50.850970 6413 tablet_bootstrap.cc:654] T ce9baa646d304268886d2cb76294c47c P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.851336 329 tablet_service.cc:1467] Processing CreateTablet for tablet 7449ead7db7b4f7083aa4b7358fea0de (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=44df1eb888434365abd5f82ea84af4e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:50.851603 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7449ead7db7b4f7083aa4b7358fea0de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.853040 6749 tablet_bootstrap.cc:492] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:50.853955 6751 tablet_bootstrap.cc:492] T 7449ead7db7b4f7083aa4b7358fea0de P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:50.854055 6749 tablet_bootstrap.cc:654] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.854470 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 0cc8946505474737a610de35ddf0f2f6 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=44df1eb888434365abd5f82ea84af4e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:50.854741 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cc8946505474737a610de35ddf0f2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.854741 6504 tablet_bootstrap.cc:492] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:50.854826 6504 ts_tablet_manager.cc:1367] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:50.855289 6504 raft_consensus.cc:348] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:50.855404 6504 raft_consensus.cc:374] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.855432 6504 raft_consensus.cc:729] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.855481 6751 tablet_bootstrap.cc:654] T 7449ead7db7b4f7083aa4b7358fea0de P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.855537 6504 consensus_queue.cc:260] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:50.855827 6504 ts_tablet_manager.cc:1398] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:50.856575 6751 tablet_bootstrap.cc:492] T 7449ead7db7b4f7083aa4b7358fea0de P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:50.856585 6746 tablet_bootstrap.cc:492] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:50.856662 6751 ts_tablet_manager.cc:1367] T 7449ead7db7b4f7083aa4b7358fea0de P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:50.857075 6751 raft_consensus.cc:348] T 7449ead7db7b4f7083aa4b7358fea0de P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:50.857174 6751 raft_consensus.cc:374] T 7449ead7db7b4f7083aa4b7358fea0de P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.857211 6751 raft_consensus.cc:729] T 7449ead7db7b4f7083aa4b7358fea0de P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.857342 6751 consensus_queue.cc:260] T 7449ead7db7b4f7083aa4b7358fea0de P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:50.857595 6503 tablet_bootstrap.cc:492] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:50.857668 6751 ts_tablet_manager.cc:1398] T 7449ead7db7b4f7083aa4b7358fea0de P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:50.857283 6747 tablet_bootstrap.cc:492] T 7449ead7db7b4f7083aa4b7358fea0de P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:50.858541 6503 tablet_bootstrap.cc:654] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.859099 6413 tablet_bootstrap.cc:492] T ce9baa646d304268886d2cb76294c47c P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:50.859182 6413 ts_tablet_manager.cc:1367] T ce9baa646d304268886d2cb76294c47c P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240501 14:09:50.859639 6413 raft_consensus.cc:348] T ce9baa646d304268886d2cb76294c47c P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:50.859740 6413 raft_consensus.cc:374] T ce9baa646d304268886d2cb76294c47c P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.859771 6413 raft_consensus.cc:729] T ce9baa646d304268886d2cb76294c47c P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.859896 6413 consensus_queue.cc:260] T ce9baa646d304268886d2cb76294c47c P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:50.860186 6413 ts_tablet_manager.cc:1398] T ce9baa646d304268886d2cb76294c47c P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:50.860375 6413 tablet_bootstrap.cc:492] T 0cc8946505474737a610de35ddf0f2f6 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:50.861382 6413 tablet_bootstrap.cc:654] T 0cc8946505474737a610de35ddf0f2f6 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.864779 6749 tablet_bootstrap.cc:492] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:50.864914 6749 ts_tablet_manager.cc:1367] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:50.864166 6753 tablet_bootstrap.cc:492] T ce9baa646d304268886d2cb76294c47c P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:50.865502 6503 tablet_bootstrap.cc:492] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:50.865579 6503 ts_tablet_manager.cc:1367] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:50.865489 6749 raft_consensus.cc:348] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:50.865635 6749 raft_consensus.cc:374] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.865715 6749 raft_consensus.cc:729] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.865906 6749 consensus_queue.cc:260] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:50.866050 6503 raft_consensus.cc:348] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:50.866156 6503 raft_consensus.cc:374] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.866189 6503 raft_consensus.cc:729] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.866318 6746 tablet_bootstrap.cc:654] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.866300 6503 consensus_queue.cc:260] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:50.866541 6747 tablet_bootstrap.cc:654] T 7449ead7db7b4f7083aa4b7358fea0de P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.866580 6503 ts_tablet_manager.cc:1398] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:50.867034 6749 ts_tablet_manager.cc:1398] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:50.868433 6746 tablet_bootstrap.cc:492] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:50.868575 6746 ts_tablet_manager.cc:1367] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:50.868778 6413 tablet_bootstrap.cc:492] T 0cc8946505474737a610de35ddf0f2f6 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:50.868860 6413 ts_tablet_manager.cc:1367] T 0cc8946505474737a610de35ddf0f2f6 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:50.869046 6746 raft_consensus.cc:348] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:50.869159 6746 raft_consensus.cc:374] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.869187 6746 raft_consensus.cc:729] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.869334 6413 raft_consensus.cc:348] T 0cc8946505474737a610de35ddf0f2f6 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:50.869338 6746 consensus_queue.cc:260] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:50.869426 6413 raft_consensus.cc:374] T 0cc8946505474737a610de35ddf0f2f6 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.869455 6413 raft_consensus.cc:729] T 0cc8946505474737a610de35ddf0f2f6 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.869560 6413 consensus_queue.cc:260] T 0cc8946505474737a610de35ddf0f2f6 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:50.869697 6746 ts_tablet_manager.cc:1398] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:50.869809 6413 ts_tablet_manager.cc:1398] T 0cc8946505474737a610de35ddf0f2f6 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:50.870620 6753 tablet_bootstrap.cc:654] T ce9baa646d304268886d2cb76294c47c P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.871879 6753 tablet_bootstrap.cc:492] T ce9baa646d304268886d2cb76294c47c P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:50.871994 6753 ts_tablet_manager.cc:1367] T ce9baa646d304268886d2cb76294c47c P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:50.872445 6753 raft_consensus.cc:348] T ce9baa646d304268886d2cb76294c47c P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:50.872570 6753 raft_consensus.cc:374] T ce9baa646d304268886d2cb76294c47c P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.872617 6753 raft_consensus.cc:729] T ce9baa646d304268886d2cb76294c47c P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.872745 6753 consensus_queue.cc:260] T ce9baa646d304268886d2cb76294c47c P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:50.873011 6753 ts_tablet_manager.cc:1398] T ce9baa646d304268886d2cb76294c47c P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240501 14:09:50.879418 30455 tablet.cc:2369] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:50.882583 6747 tablet_bootstrap.cc:492] T 7449ead7db7b4f7083aa4b7358fea0de P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:50.882705 6747 ts_tablet_manager.cc:1367] T 7449ead7db7b4f7083aa4b7358fea0de P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240501 14:09:50.883152 6747 raft_consensus.cc:348] T 7449ead7db7b4f7083aa4b7358fea0de P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:50.883297 6747 raft_consensus.cc:374] T 7449ead7db7b4f7083aa4b7358fea0de P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.883352 6747 raft_consensus.cc:729] T 7449ead7db7b4f7083aa4b7358fea0de P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.884214 6747 consensus_queue.cc:260] T 7449ead7db7b4f7083aa4b7358fea0de P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:50.884678 6747 ts_tablet_manager.cc:1398] T 7449ead7db7b4f7083aa4b7358fea0de P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:50.894762 29873 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:50.894888 29873 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:50.897471 29873 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 60096eb27894484190bd5a3a23f03e68
I20240501 14:09:50.897985 30388 tablet_service.cc:1514] Processing DeleteTablet for tablet 60096eb27894484190bd5a3a23f03e68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:50 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:50.898689 6771 tablet_replica.cc:330] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:50.898855 32533 tablet_service.cc:1514] Processing DeleteTablet for tablet 60096eb27894484190bd5a3a23f03e68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:50 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:50.898895 6771 raft_consensus.cc:2229] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:50.899109 6771 raft_consensus.cc:2258] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:50.899104 32665 tablet_service.cc:1514] Processing DeleteTablet for tablet 60096eb27894484190bd5a3a23f03e68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:50 UTC) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:50.899294 6617 tablet_replica.cc:330] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:50.899431 6617 raft_consensus.cc:2229] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:50.899544 6617 raft_consensus.cc:2258] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:50.899693 6771 ts_tablet_manager.cc:1875] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:50.900285 6617 ts_tablet_manager.cc:1875] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:50.901368 6772 tablet_replica.cc:330] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:50.901584 6772 raft_consensus.cc:2229] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:50.901804 6771 ts_tablet_manager.cc:1888] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:50.901893 6771 log.cc:1196] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/60096eb27894484190bd5a3a23f03e68
I20240501 14:09:50.901886 6772 raft_consensus.cc:2258] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:50.903307 6771 ts_tablet_manager.cc:1909] T 60096eb27894484190bd5a3a23f03e68 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:50.903955 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 60096eb27894484190bd5a3a23f03e68 (table rebalancer_dynamic_table_new_name [id=2ae45ec6a8f0431181d15f77d852cb3c]) successfully deleted
I20240501 14:09:50.904727 6617 ts_tablet_manager.cc:1888] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:50.905328 6617 log.cc:1196] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/60096eb27894484190bd5a3a23f03e68
I20240501 14:09:50.907526 6772 ts_tablet_manager.cc:1875] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:50.908465 6617 ts_tablet_manager.cc:1909] T 60096eb27894484190bd5a3a23f03e68 P db935501e9104de5b15c8780681d6c38: Deleting consensus metadata
I20240501 14:09:50.909986 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet 60096eb27894484190bd5a3a23f03e68 (table rebalancer_dynamic_table_new_name [id=2ae45ec6a8f0431181d15f77d852cb3c]) successfully deleted
I20240501 14:09:50.910310 6772 ts_tablet_manager.cc:1888] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:50.910429 6772 log.cc:1196] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/60096eb27894484190bd5a3a23f03e68
I20240501 14:09:50.910864 6772 ts_tablet_manager.cc:1909] T 60096eb27894484190bd5a3a23f03e68 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:50.911563 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet 60096eb27894484190bd5a3a23f03e68 (table rebalancer_dynamic_table_new_name [id=2ae45ec6a8f0431181d15f77d852cb3c]) successfully deleted
I20240501 14:09:50.911878 329 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:50.917621 32533 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:50.922900 6666 raft_consensus.cc:482] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:50.922994 6666 raft_consensus.cc:504] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:50.923424 6666 leader_election.cc:288] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:50.923802 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7449ead7db7b4f7083aa4b7358fea0de" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:50.923926 30277 raft_consensus.cc:2454] T 7449ead7db7b4f7083aa4b7358fea0de P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:50.924018 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7449ead7db7b4f7083aa4b7358fea0de" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:50.924150 349 raft_consensus.cc:2454] T 7449ead7db7b4f7083aa4b7358fea0de P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:50.924404 30081 leader_election.cc:302] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:50.924762 6666 raft_consensus.cc:2781] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:50.924824 6666 raft_consensus.cc:482] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:50.924867 6666 raft_consensus.cc:3037] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.926210 6666 raft_consensus.cc:504] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:50.926532 6666 leader_election.cc:288] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:50.926925 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7449ead7db7b4f7083aa4b7358fea0de" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:50.927031 349 raft_consensus.cc:3037] T 7449ead7db7b4f7083aa4b7358fea0de P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.928025 349 raft_consensus.cc:2454] T 7449ead7db7b4f7083aa4b7358fea0de P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:50.928355 30081 leader_election.cc:302] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:50.928525 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7449ead7db7b4f7083aa4b7358fea0de" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:50.928635 30277 raft_consensus.cc:3037] T 7449ead7db7b4f7083aa4b7358fea0de P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.929601 30277 raft_consensus.cc:2454] T 7449ead7db7b4f7083aa4b7358fea0de P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:50.930529 6666 raft_consensus.cc:2781] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:50.931114 6666 raft_consensus.cc:686] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:50.931320 6666 consensus_queue.cc:237] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:50.931665 30126 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:50.931927 29993 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:50.934036 30388 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:50.935385 29870 catalog_manager.cc:5484] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:50.938495 30255 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:50.944365 32665 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240501 14:09:50.953790 30324 tablet.cc:2369] T 7449ead7db7b4f7083aa4b7358fea0de P 1d0bb69611af4876a2b3b33c4766e13a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:50.959890 32401 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 14:09:50.994360 30193 tablet.cc:2369] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:51.000517 29870 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:51.006916 30126 tablet_service.cc:1467] Processing CreateTablet for tablet fef54196bc934976910db4cdfee9a2ae (DEFAULT_TABLE table=rebalancer_dynamic_table [id=714c415375564eb8a432c8a903aeab12]), partition=
I20240501 14:09:51.007181 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef54196bc934976910db4cdfee9a2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.007889 29993 tablet_service.cc:1467] Processing CreateTablet for tablet fef54196bc934976910db4cdfee9a2ae (DEFAULT_TABLE table=rebalancer_dynamic_table [id=714c415375564eb8a432c8a903aeab12]), partition=
I20240501 14:09:51.008109 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef54196bc934976910db4cdfee9a2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.009898 6746 tablet_bootstrap.cc:492] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:51.010510 6749 tablet_bootstrap.cc:492] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:51.010823 6746 tablet_bootstrap.cc:654] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.010926 329 tablet_service.cc:1467] Processing CreateTablet for tablet fef54196bc934976910db4cdfee9a2ae (DEFAULT_TABLE table=rebalancer_dynamic_table [id=714c415375564eb8a432c8a903aeab12]), partition=
I20240501 14:09:51.011169 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef54196bc934976910db4cdfee9a2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.011422 6749 tablet_bootstrap.cc:654] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.012226 6746 tablet_bootstrap.cc:492] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:51.012374 6746 ts_tablet_manager.cc:1367] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:51.012799 6746 raft_consensus.cc:348] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:51.012876 6746 raft_consensus.cc:374] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.012897 6746 raft_consensus.cc:729] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.013007 6746 consensus_queue.cc:260] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:51.013293 6746 ts_tablet_manager.cc:1398] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:51.013746 6751 tablet_bootstrap.cc:492] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:51.014389 6751 tablet_bootstrap.cc:654] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.014832 6749 tablet_bootstrap.cc:492] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:51.014905 6749 ts_tablet_manager.cc:1367] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:51.015331 6749 raft_consensus.cc:348] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:51.015421 6749 raft_consensus.cc:374] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.015450 6749 raft_consensus.cc:729] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.015568 6749 consensus_queue.cc:260] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:51.015830 6749 ts_tablet_manager.cc:1398] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:51.017037 6751 tablet_bootstrap.cc:492] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:51.017128 6751 ts_tablet_manager.cc:1367] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:51.017472 5552 raft_consensus.cc:482] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:51.017563 5552 raft_consensus.cc:504] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:51.017622 6751 raft_consensus.cc:348] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:51.017719 6751 raft_consensus.cc:374] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.017741 6751 raft_consensus.cc:729] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.017832 6751 consensus_queue.cc:260] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:51.017882 5552 leader_election.cc:288] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:51.018117 6751 ts_tablet_manager.cc:1398] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:51.018298 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cc8946505474737a610de35ddf0f2f6" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:51.018414 30406 raft_consensus.cc:2454] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:51.018688 32487 leader_election.cc:302] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:51.018847 5552 raft_consensus.cc:2781] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:51.018899 5552 raft_consensus.cc:482] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:51.018931 5552 raft_consensus.cc:3037] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.019114 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cc8946505474737a610de35ddf0f2f6" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:51.019241 32685 raft_consensus.cc:2454] T 0cc8946505474737a610de35ddf0f2f6 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:51.019927 5552 raft_consensus.cc:504] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:51.020332 5552 leader_election.cc:288] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:51.020953 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cc8946505474737a610de35ddf0f2f6" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:51.021101 30406 raft_consensus.cc:3037] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.021548 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cc8946505474737a610de35ddf0f2f6" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:51.021643 32685 raft_consensus.cc:3037] T 0cc8946505474737a610de35ddf0f2f6 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.022178 30406 raft_consensus.cc:2454] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:51.022666 32685 raft_consensus.cc:2454] T 0cc8946505474737a610de35ddf0f2f6 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:51.022910 32487 leader_election.cc:302] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:51.023280 5552 raft_consensus.cc:2781] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:51.023370 5552 raft_consensus.cc:686] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:09:51.023505 5552 consensus_queue.cc:237] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:51.026579 29870 catalog_manager.cc:5484] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
W20240501 14:09:51.059549 30062 tablet.cc:2369] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:51.080842 6075 raft_consensus.cc:482] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:51.080997 6075 raft_consensus.cc:504] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:51.081380 6075 leader_election.cc:288] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:51.081691 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef54196bc934976910db4cdfee9a2ae" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:51.081833 30146 raft_consensus.cc:2454] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:51.081920 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef54196bc934976910db4cdfee9a2ae" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:51.082091 349 raft_consensus.cc:2454] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:51.082165 29951 leader_election.cc:302] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:51.082383 6075 raft_consensus.cc:2781] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:51.082449 6075 raft_consensus.cc:482] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:51.082491 6075 raft_consensus.cc:3037] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.083667 6075 raft_consensus.cc:504] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:51.083946 6075 leader_election.cc:288] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:51.084476 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef54196bc934976910db4cdfee9a2ae" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:51.084591 30146 raft_consensus.cc:3037] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.085651 30146 raft_consensus.cc:2454] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:51.085883 29951 leader_election.cc:302] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:51.085897 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef54196bc934976910db4cdfee9a2ae" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:51.086004 349 raft_consensus.cc:3037] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.086061 6075 raft_consensus.cc:2781] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:51.086159 6075 raft_consensus.cc:686] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:51.086298 6075 consensus_queue.cc:237] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:51.087011 349 raft_consensus.cc:2454] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:51.088555 29869 catalog_manager.cc:5484] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:51.176086 6819 raft_consensus.cc:482] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:51.176203 6819 raft_consensus.cc:504] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:51.176635 6819 leader_election.cc:288] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:51.176951 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9baa646d304268886d2cb76294c47c" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:51.176991 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9baa646d304268886d2cb76294c47c" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:51.177083 32415 raft_consensus.cc:2454] T ce9baa646d304268886d2cb76294c47c P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:51.177116 32685 raft_consensus.cc:2454] T ce9baa646d304268886d2cb76294c47c P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:51.177376 29950 leader_election.cc:302] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:51.177569 6819 raft_consensus.cc:2781] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:51.177639 6819 raft_consensus.cc:482] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:51.177695 6819 raft_consensus.cc:3037] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.178690 6819 raft_consensus.cc:504] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:51.178979 6819 leader_election.cc:288] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:51.179248 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9baa646d304268886d2cb76294c47c" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:51.179356 32415 raft_consensus.cc:3037] T ce9baa646d304268886d2cb76294c47c P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.179394 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9baa646d304268886d2cb76294c47c" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:51.179522 32685 raft_consensus.cc:3037] T ce9baa646d304268886d2cb76294c47c P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.180135 32415 raft_consensus.cc:2454] T ce9baa646d304268886d2cb76294c47c P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:51.180282 32685 raft_consensus.cc:2454] T ce9baa646d304268886d2cb76294c47c P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:51.180409 29950 leader_election.cc:302] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:51.180596 6819 raft_consensus.cc:2781] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:51.180711 6819 raft_consensus.cc:686] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:51.180861 6819 consensus_queue.cc:237] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:51.183399 29873 catalog_manager.cc:5484] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:51.199599 29866 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:51.207307 30146 raft_consensus.cc:1261] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:51.208356 349 raft_consensus.cc:1261] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:51.208478 6075 consensus_queue.cc:1035] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:51.209251 6075 consensus_queue.cc:1035] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:51.211591 29873 catalog_manager.cc:5576] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457) does not have the latest schema for tablet fef54196bc934976910db4cdfee9a2ae (table rebalancer_dynamic_table [id=714c415375564eb8a432c8a903aeab12]). Expected version 1 got 0
I20240501 14:09:51.212396 29872 catalog_manager.cc:5576] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937) does not have the latest schema for tablet fef54196bc934976910db4cdfee9a2ae (table rebalancer_dynamic_table [id=714c415375564eb8a432c8a903aeab12]). Expected version 1 got 0
I20240501 14:09:51.217293 29993 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:51.217481 29872 catalog_manager.cc:5576] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457) does not have the latest schema for tablet fef54196bc934976910db4cdfee9a2ae (table rebalancer_dynamic_table [id=714c415375564eb8a432c8a903aeab12]). Expected version 1 got 0
I20240501 14:09:51.219632 6842 tablet.cc:1722] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:51.219949 6842 tablet.cc:1620] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3: MemRowSet was empty: no flush needed.
I20240501 14:09:51.223207 29872 catalog_manager.cc:5576] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937) does not have the latest schema for tablet fef54196bc934976910db4cdfee9a2ae (table rebalancer_dynamic_table [id=714c415375564eb8a432c8a903aeab12]). Expected version 1 got 0
I20240501 14:09:51.223968 6843 ts_tablet_manager.cc:902] T 529f938305214c3ba33f61e06d1acd77 P db935501e9104de5b15c8780681d6c38: Initiating tablet copy from peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:51.224305 6839 tablet.cc:1722] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:51.224532 6843 tablet_copy_client.cc:308] T 529f938305214c3ba33f61e06d1acd77 P db935501e9104de5b15c8780681d6c38: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.130:36457
I20240501 14:09:51.225167 6839 tablet.cc:1620] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e: MemRowSet was empty: no flush needed.
I20240501 14:09:51.225579 30165 tablet_copy_service.cc:140] P d35c16dcb506426993811bb2519d7db3: Received BeginTabletCopySession request for tablet 529f938305214c3ba33f61e06d1acd77 from peer db935501e9104de5b15c8780681d6c38 ({username='slave'} at 127.22.252.135:49787)
I20240501 14:09:51.225672 30165 tablet_copy_service.cc:161] P d35c16dcb506426993811bb2519d7db3: Beginning new tablet copy session on tablet 529f938305214c3ba33f61e06d1acd77 from peer db935501e9104de5b15c8780681d6c38 at {username='slave'} at 127.22.252.135:49787: session id = db935501e9104de5b15c8780681d6c38-529f938305214c3ba33f61e06d1acd77
I20240501 14:09:51.226645 30165 tablet_copy_source_session.cc:205] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:51.228258 6843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 529f938305214c3ba33f61e06d1acd77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.230687 6843 tablet_copy_client.cc:783] T 529f938305214c3ba33f61e06d1acd77 P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:51.231066 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=714c415375564eb8a432c8a903aeab12] alter complete (version 1)
I20240501 14:09:51.231787 32533 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:51.230835 6843 tablet_copy_client.cc:647] T 529f938305214c3ba33f61e06d1acd77 P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:51.229575 6844 tablet.cc:1722] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:51.232833 6844 tablet.cc:1620] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:51.241722 6843 tablet_copy_client.cc:523] T 529f938305214c3ba33f61e06d1acd77 P db935501e9104de5b15c8780681d6c38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:51.242566 30255 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:51.244359 32401 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:51.247232 329 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:51.248698 6843 tablet_bootstrap.cc:492] T 529f938305214c3ba33f61e06d1acd77 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:51.260286 32665 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:51.271862 6840 ts_tablet_manager.cc:902] T f2bc06e76e2a41429d5625ebf99e5e5c P 5f8a0cfdfdca4148a75686dc88059cf3: Initiating tablet copy from peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:51.272418 6840 tablet_copy_client.cc:308] T f2bc06e76e2a41429d5625ebf99e5e5c P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.131:43103
I20240501 14:09:51.272637 6841 ts_tablet_manager.cc:902] T 19f203c753ef481dafc091ea2d60f56f P 5f8a0cfdfdca4148a75686dc88059cf3: Initiating tablet copy from peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:51.273042 30297 tablet_copy_service.cc:140] P 1d0bb69611af4876a2b3b33c4766e13a: Received BeginTabletCopySession request for tablet f2bc06e76e2a41429d5625ebf99e5e5c from peer 5f8a0cfdfdca4148a75686dc88059cf3 ({username='slave'} at 127.22.252.134:55213)
I20240501 14:09:51.273130 30297 tablet_copy_service.cc:161] P 1d0bb69611af4876a2b3b33c4766e13a: Beginning new tablet copy session on tablet f2bc06e76e2a41429d5625ebf99e5e5c from peer 5f8a0cfdfdca4148a75686dc88059cf3 at {username='slave'} at 127.22.252.134:55213: session id = 5f8a0cfdfdca4148a75686dc88059cf3-f2bc06e76e2a41429d5625ebf99e5e5c
I20240501 14:09:51.274019 30297 tablet_copy_source_session.cc:205] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:51.274153 30388 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:51.274961 6841 tablet_copy_client.cc:308] T 19f203c753ef481dafc091ea2d60f56f P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.131:43103
I20240501 14:09:51.275365 30297 tablet_copy_service.cc:140] P 1d0bb69611af4876a2b3b33c4766e13a: Received BeginTabletCopySession request for tablet 19f203c753ef481dafc091ea2d60f56f from peer 5f8a0cfdfdca4148a75686dc88059cf3 ({username='slave'} at 127.22.252.134:55213)
I20240501 14:09:51.275442 30297 tablet_copy_service.cc:161] P 1d0bb69611af4876a2b3b33c4766e13a: Beginning new tablet copy session on tablet 19f203c753ef481dafc091ea2d60f56f from peer 5f8a0cfdfdca4148a75686dc88059cf3 at {username='slave'} at 127.22.252.134:55213: session id = 5f8a0cfdfdca4148a75686dc88059cf3-19f203c753ef481dafc091ea2d60f56f
I20240501 14:09:51.276189 30297 tablet_copy_source_session.cc:205] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:51.278766 6841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f203c753ef481dafc091ea2d60f56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.281193 6841 tablet_copy_client.cc:783] T 19f203c753ef481dafc091ea2d60f56f P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:51.281559 6841 tablet_copy_client.cc:647] T 19f203c753ef481dafc091ea2d60f56f P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:51.281862 6840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2bc06e76e2a41429d5625ebf99e5e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.283447 6840 tablet_copy_client.cc:783] T f2bc06e76e2a41429d5625ebf99e5e5c P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:51.283720 6840 tablet_copy_client.cc:647] T f2bc06e76e2a41429d5625ebf99e5e5c P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:51.286434 6840 tablet_copy_client.cc:523] T f2bc06e76e2a41429d5625ebf99e5e5c P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:51.287525 6840 tablet_bootstrap.cc:492] T f2bc06e76e2a41429d5625ebf99e5e5c P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:51.288779 6841 tablet_copy_client.cc:523] T 19f203c753ef481dafc091ea2d60f56f P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:51.289243 6843 tablet_bootstrap.cc:492] T 529f938305214c3ba33f61e06d1acd77 P db935501e9104de5b15c8780681d6c38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:51.289577 6843 tablet_bootstrap.cc:492] T 529f938305214c3ba33f61e06d1acd77 P db935501e9104de5b15c8780681d6c38: Bootstrap complete.
I20240501 14:09:51.289808 6843 ts_tablet_manager.cc:1367] T 529f938305214c3ba33f61e06d1acd77 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.041s user 0.023s sys 0.003s
I20240501 14:09:51.290123 6841 tablet_bootstrap.cc:492] T 19f203c753ef481dafc091ea2d60f56f P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:51.290398 6843 raft_consensus.cc:348] T 529f938305214c3ba33f61e06d1acd77 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:51.290594 6843 raft_consensus.cc:729] T 529f938305214c3ba33f61e06d1acd77 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.290764 6843 consensus_queue.cc:260] T 529f938305214c3ba33f61e06d1acd77 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:51.291163 6843 ts_tablet_manager.cc:1398] T 529f938305214c3ba33f61e06d1acd77 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:51.292599 30165 tablet_copy_service.cc:342] P d35c16dcb506426993811bb2519d7db3: Request end of tablet copy session db935501e9104de5b15c8780681d6c38-529f938305214c3ba33f61e06d1acd77 received from {username='slave'} at 127.22.252.135:49787
I20240501 14:09:51.292675 30165 tablet_copy_service.cc:433] P d35c16dcb506426993811bb2519d7db3: ending tablet copy session db935501e9104de5b15c8780681d6c38-529f938305214c3ba33f61e06d1acd77 on tablet 529f938305214c3ba33f61e06d1acd77 with peer db935501e9104de5b15c8780681d6c38
I20240501 14:09:51.299583 30126 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:51.331794 6841 tablet_bootstrap.cc:492] T 19f203c753ef481dafc091ea2d60f56f P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:51.332175 6841 tablet_bootstrap.cc:492] T 19f203c753ef481dafc091ea2d60f56f P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap complete.
I20240501 14:09:51.332393 6841 ts_tablet_manager.cc:1367] T 19f203c753ef481dafc091ea2d60f56f P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.042s user 0.022s sys 0.004s
I20240501 14:09:51.332811 6841 raft_consensus.cc:348] T 19f203c753ef481dafc091ea2d60f56f P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:51.332923 6841 raft_consensus.cc:729] T 19f203c753ef481dafc091ea2d60f56f P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.333066 6841 consensus_queue.cc:260] T 19f203c753ef481dafc091ea2d60f56f P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:51.333483 6841 ts_tablet_manager.cc:1398] T 19f203c753ef481dafc091ea2d60f56f P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:51.333936 30297 tablet_copy_service.cc:342] P 1d0bb69611af4876a2b3b33c4766e13a: Request end of tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-19f203c753ef481dafc091ea2d60f56f received from {username='slave'} at 127.22.252.134:55213
I20240501 14:09:51.334008 30297 tablet_copy_service.cc:433] P 1d0bb69611af4876a2b3b33c4766e13a: ending tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-19f203c753ef481dafc091ea2d60f56f on tablet 19f203c753ef481dafc091ea2d60f56f with peer 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:09:51.337939 29872 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:51.339406 6840 tablet_bootstrap.cc:492] T f2bc06e76e2a41429d5625ebf99e5e5c P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:51.339700 6840 tablet_bootstrap.cc:492] T f2bc06e76e2a41429d5625ebf99e5e5c P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap complete.
I20240501 14:09:51.339917 6840 ts_tablet_manager.cc:1367] T f2bc06e76e2a41429d5625ebf99e5e5c P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.052s user 0.024s sys 0.000s
I20240501 14:09:51.340325 6840 raft_consensus.cc:348] T f2bc06e76e2a41429d5625ebf99e5e5c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:51.340495 6840 raft_consensus.cc:729] T f2bc06e76e2a41429d5625ebf99e5e5c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.340662 6840 consensus_queue.cc:260] T f2bc06e76e2a41429d5625ebf99e5e5c P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:51.341164 6840 ts_tablet_manager.cc:1398] T f2bc06e76e2a41429d5625ebf99e5e5c P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:51.342691 30297 tablet_copy_service.cc:342] P 1d0bb69611af4876a2b3b33c4766e13a: Request end of tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-f2bc06e76e2a41429d5625ebf99e5e5c received from {username='slave'} at 127.22.252.134:55213
I20240501 14:09:51.342767 30297 tablet_copy_service.cc:433] P 1d0bb69611af4876a2b3b33c4766e13a: ending tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-f2bc06e76e2a41429d5625ebf99e5e5c on tablet f2bc06e76e2a41429d5625ebf99e5e5c with peer 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:09:51.345458 6839 tablet.cc:1722] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:51.345782 6839 tablet.cc:1620] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e: MemRowSet was empty: no flush needed.
I20240501 14:09:51.345902 6842 tablet.cc:1722] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:51.345906 6844 tablet.cc:1722] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:51.346192 6842 tablet.cc:1620] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3: MemRowSet was empty: no flush needed.
I20240501 14:09:51.346235 6844 tablet.cc:1620] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:51.351868 29866 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=714c415375564eb8a432c8a903aeab12] alter complete (version 2)
I20240501 14:09:51.367863 6666 consensus_queue.cc:1035] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:51.370741 6580 consensus_queue.cc:1035] T 7449ead7db7b4f7083aa4b7358fea0de P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:51.453850 29870 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:51.459723 29993 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:51.469806 329 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:51.470106 6839 tablet.cc:1722] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:51.471685 6842 tablet.cc:1722] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:51.472251 32533 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:51.477697 6844 tablet.cc:1722] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:51.477802 30255 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:51.480140 32665 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:51.481349 29867 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=714c415375564eb8a432c8a903aeab12] alter complete (version 3)
I20240501 14:09:51.487071 30126 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:51.496109 30388 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:51.515776 32401 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:51.556047 6456 consensus_queue.cc:1035] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:51.560452 6929 consensus_queue.cc:1035] T 0cc8946505474737a610de35ddf0f2f6 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:51.587582 29869 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:51.587728 29869 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:51.591456 29869 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet fef54196bc934976910db4cdfee9a2ae
I20240501 14:09:51.592080 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet fef54196bc934976910db4cdfee9a2ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:51 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:51.592255 30126 tablet_service.cc:1514] Processing DeleteTablet for tablet fef54196bc934976910db4cdfee9a2ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:51 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:51.592515 6965 tablet_replica.cc:330] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:51.592680 6965 raft_consensus.cc:2229] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:51.592866 6965 raft_consensus.cc:2258] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:51.593758 6965 ts_tablet_manager.cc:1875] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:51.594025 329 tablet_service.cc:1514] Processing DeleteTablet for tablet fef54196bc934976910db4cdfee9a2ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:51 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:51.597034 6965 ts_tablet_manager.cc:1888] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:51.597165 6965 log.cc:1196] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/fef54196bc934976910db4cdfee9a2ae
I20240501 14:09:51.597556 6965 ts_tablet_manager.cc:1909] T fef54196bc934976910db4cdfee9a2ae P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:51.597709 6075 consensus_queue.cc:1035] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:51.598889 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet fef54196bc934976910db4cdfee9a2ae (table rebalancer_dynamic_table_new_name [id=714c415375564eb8a432c8a903aeab12]) successfully deleted
I20240501 14:09:51.600704 6593 consensus_queue.cc:1035] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:51.603727 6966 tablet_replica.cc:330] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:51.603982 6966 raft_consensus.cc:2229] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:51.604153 6966 raft_consensus.cc:2258] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:51.606498 6966 ts_tablet_manager.cc:1875] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:51.609439 6966 ts_tablet_manager.cc:1888] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:51.609556 6966 log.cc:1196] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/fef54196bc934976910db4cdfee9a2ae
I20240501 14:09:51.610033 6966 ts_tablet_manager.cc:1909] T fef54196bc934976910db4cdfee9a2ae P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:09:51.610666 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet fef54196bc934976910db4cdfee9a2ae (table rebalancer_dynamic_table_new_name [id=714c415375564eb8a432c8a903aeab12]) successfully deleted
I20240501 14:09:51.629046 6964 tablet_replica.cc:330] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:51.629271 6964 raft_consensus.cc:2229] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:51.629541 6964 raft_consensus.cc:2258] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:51.632776 6964 ts_tablet_manager.cc:1875] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:51.637637 6964 ts_tablet_manager.cc:1888] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:51.637713 6964 log.cc:1196] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/fef54196bc934976910db4cdfee9a2ae
I20240501 14:09:51.638101 6964 ts_tablet_manager.cc:1909] T fef54196bc934976910db4cdfee9a2ae P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:51.638769 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet fef54196bc934976910db4cdfee9a2ae (table rebalancer_dynamic_table_new_name [id=714c415375564eb8a432c8a903aeab12]) successfully deleted
I20240501 14:09:51.654711 32553 raft_consensus.cc:1203] T 19f203c753ef481dafc091ea2d60f56f P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:51.687891 29873 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:51.694577 32401 tablet_service.cc:1467] Processing CreateTablet for tablet 89ce3093e24347f3a4e121af452c6361 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4da773d0a24f4f5d9ec6e0b50c97d669]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.694872 32401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ce3093e24347f3a4e121af452c6361. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.694802 32533 tablet_service.cc:1467] Processing CreateTablet for tablet 89ce3093e24347f3a4e121af452c6361 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4da773d0a24f4f5d9ec6e0b50c97d669]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.695106 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ce3093e24347f3a4e121af452c6361. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.696969 329 tablet_service.cc:1467] Processing CreateTablet for tablet 6208e4b3d25941848b2ca533edd5a7b9 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4da773d0a24f4f5d9ec6e0b50c97d669]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.697247 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6208e4b3d25941848b2ca533edd5a7b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.698575 7011 tablet_bootstrap.cc:492] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:51.699546 7011 tablet_bootstrap.cc:654] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.699565 7013 tablet_bootstrap.cc:492] T 6208e4b3d25941848b2ca533edd5a7b9 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:51.699709 327 tablet_service.cc:1467] Processing CreateTablet for tablet ea69585d641342b8a1340339bb2732b3 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4da773d0a24f4f5d9ec6e0b50c97d669]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.699927 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea69585d641342b8a1340339bb2732b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.701020 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 89ce3093e24347f3a4e121af452c6361 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4da773d0a24f4f5d9ec6e0b50c97d669]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.701301 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ce3093e24347f3a4e121af452c6361. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.701632 7013 tablet_bootstrap.cc:654] T 6208e4b3d25941848b2ca533edd5a7b9 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.703305 32685 raft_consensus.cc:1203] T 529f938305214c3ba33f61e06d1acd77 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:51.703480 7016 tablet_bootstrap.cc:492] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:51.704615 7016 tablet_bootstrap.cc:654] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.704834 32533 tablet_service.cc:1467] Processing CreateTablet for tablet ea69585d641342b8a1340339bb2732b3 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4da773d0a24f4f5d9ec6e0b50c97d669]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.705103 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea69585d641342b8a1340339bb2732b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.705207 7012 tablet_bootstrap.cc:492] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:51.705971 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 6208e4b3d25941848b2ca533edd5a7b9 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4da773d0a24f4f5d9ec6e0b50c97d669]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.705964 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 6208e4b3d25941848b2ca533edd5a7b9 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4da773d0a24f4f5d9ec6e0b50c97d669]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.706161 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6208e4b3d25941848b2ca533edd5a7b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.706178 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6208e4b3d25941848b2ca533edd5a7b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.708873 29873 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:51.711454 32665 tablet_service.cc:1467] Processing CreateTablet for tablet ea69585d641342b8a1340339bb2732b3 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4da773d0a24f4f5d9ec6e0b50c97d669]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.711700 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea69585d641342b8a1340339bb2732b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.715353 7012 tablet_bootstrap.cc:654] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.716269 7018 tablet_bootstrap.cc:492] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:51.717660 7013 tablet_bootstrap.cc:492] T 6208e4b3d25941848b2ca533edd5a7b9 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:51.717809 7013 ts_tablet_manager.cc:1367] T 6208e4b3d25941848b2ca533edd5a7b9 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 14:09:51.718459 7013 raft_consensus.cc:348] T 6208e4b3d25941848b2ca533edd5a7b9 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:51.718660 7013 raft_consensus.cc:374] T 6208e4b3d25941848b2ca533edd5a7b9 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.718742 7013 raft_consensus.cc:729] T 6208e4b3d25941848b2ca533edd5a7b9 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.718951 7013 consensus_queue.cc:260] T 6208e4b3d25941848b2ca533edd5a7b9 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:51.719422 7013 ts_tablet_manager.cc:1398] T 6208e4b3d25941848b2ca533edd5a7b9 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:51.719523 7012 tablet_bootstrap.cc:492] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:51.719636 7013 tablet_bootstrap.cc:492] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:51.719672 7012 ts_tablet_manager.cc:1367] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 14:09:51.720451 7012 raft_consensus.cc:348] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:51.720626 7012 raft_consensus.cc:374] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.720726 7012 raft_consensus.cc:729] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.720870 7013 tablet_bootstrap.cc:654] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.720963 30255 tablet_service.cc:1467] Processing CreateTablet for tablet 918ddead97f64b10ac165dada9078d41 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=460b17cdb5434a0faf4015b55014b0cc]), partition=
I20240501 14:09:51.721192 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 918ddead97f64b10ac165dada9078d41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.721859 7011 tablet_bootstrap.cc:492] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:51.721946 7011 ts_tablet_manager.cc:1367] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240501 14:09:51.723807 7012 consensus_queue.cc:260] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:51.724195 7012 ts_tablet_manager.cc:1398] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:09:51.724299 327 tablet_service.cc:1467] Processing CreateTablet for tablet 918ddead97f64b10ac165dada9078d41 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=460b17cdb5434a0faf4015b55014b0cc]), partition=
I20240501 14:09:51.724517 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 918ddead97f64b10ac165dada9078d41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.727313 7018 tablet_bootstrap.cc:654] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.727530 7011 raft_consensus.cc:348] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:51.728281 7011 raft_consensus.cc:374] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.728376 7011 raft_consensus.cc:729] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.728542 7011 consensus_queue.cc:260] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:51.729601 7011 ts_tablet_manager.cc:1398] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:09:51.729843 7011 tablet_bootstrap.cc:492] T ea69585d641342b8a1340339bb2732b3 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:51.730778 7011 tablet_bootstrap.cc:654] T ea69585d641342b8a1340339bb2732b3 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.732668 7016 tablet_bootstrap.cc:492] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:51.732777 7016 ts_tablet_manager.cc:1367] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240501 14:09:51.733165 7016 raft_consensus.cc:348] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:51.733276 7016 raft_consensus.cc:374] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.733321 7016 raft_consensus.cc:729] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.733423 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 918ddead97f64b10ac165dada9078d41 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=460b17cdb5434a0faf4015b55014b0cc]), partition=
I20240501 14:09:51.733603 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 918ddead97f64b10ac165dada9078d41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.733937 7011 tablet_bootstrap.cc:492] T ea69585d641342b8a1340339bb2732b3 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:51.734035 7011 ts_tablet_manager.cc:1367] T ea69585d641342b8a1340339bb2732b3 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:51.734495 7011 raft_consensus.cc:348] T ea69585d641342b8a1340339bb2732b3 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:51.734602 7011 raft_consensus.cc:374] T ea69585d641342b8a1340339bb2732b3 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.734635 7011 raft_consensus.cc:729] T ea69585d641342b8a1340339bb2732b3 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.734758 7011 consensus_queue.cc:260] T ea69585d641342b8a1340339bb2732b3 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:51.735071 7011 ts_tablet_manager.cc:1398] T ea69585d641342b8a1340339bb2732b3 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:51.733428 7016 consensus_queue.cc:260] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:51.737658 7016 ts_tablet_manager.cc:1398] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:51.737867 7016 tablet_bootstrap.cc:492] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:51.739574 7021 tablet_bootstrap.cc:492] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:51.740887 7021 tablet_bootstrap.cc:654] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.742596 7013 tablet_bootstrap.cc:492] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:51.742682 7013 ts_tablet_manager.cc:1367] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240501 14:09:51.743139 7013 raft_consensus.cc:348] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:51.743259 7013 raft_consensus.cc:374] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.743294 7013 raft_consensus.cc:729] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.743436 7013 consensus_queue.cc:260] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:51.743703 7013 ts_tablet_manager.cc:1398] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:51.743858 7013 tablet_bootstrap.cc:492] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:51.745492 7013 tablet_bootstrap.cc:654] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.746009 7016 tablet_bootstrap.cc:654] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.746476 7013 tablet_bootstrap.cc:492] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:51.746621 7013 ts_tablet_manager.cc:1367] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:51.747162 7013 raft_consensus.cc:348] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:51.747272 7013 raft_consensus.cc:374] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.747287 7016 tablet_bootstrap.cc:492] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:51.747309 7013 raft_consensus.cc:729] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.747396 7016 ts_tablet_manager.cc:1367] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:51.747432 7013 consensus_queue.cc:260] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:51.747843 7016 raft_consensus.cc:348] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:51.747931 7016 raft_consensus.cc:374] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.747989 7013 ts_tablet_manager.cc:1398] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:51.747985 7016 raft_consensus.cc:729] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.748104 7016 consensus_queue.cc:260] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:51.748389 7016 ts_tablet_manager.cc:1398] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:51.748575 7018 tablet_bootstrap.cc:492] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:51.748663 7018 ts_tablet_manager.cc:1367] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.001s
I20240501 14:09:51.749166 7018 raft_consensus.cc:348] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:51.749274 7018 raft_consensus.cc:374] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.749318 7018 raft_consensus.cc:729] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.749429 7018 consensus_queue.cc:260] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:51.749707 7018 ts_tablet_manager.cc:1398] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:51.749879 7018 tablet_bootstrap.cc:492] T ea69585d641342b8a1340339bb2732b3 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:51.749991 7021 tablet_bootstrap.cc:492] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:51.750052 7021 ts_tablet_manager.cc:1367] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20240501 14:09:51.750371 7021 raft_consensus.cc:348] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:51.750484 7021 raft_consensus.cc:374] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.750512 7021 raft_consensus.cc:729] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.750697 7021 consensus_queue.cc:260] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:51.750847 7018 tablet_bootstrap.cc:654] T ea69585d641342b8a1340339bb2732b3 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.751155 7021 ts_tablet_manager.cc:1398] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:51.756744 7016 tablet_bootstrap.cc:492] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:51.758214 7016 tablet_bootstrap.cc:654] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.759464 7018 tablet_bootstrap.cc:492] T ea69585d641342b8a1340339bb2732b3 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:51.759382 7016 tablet_bootstrap.cc:492] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:51.759558 7018 ts_tablet_manager.cc:1367] T ea69585d641342b8a1340339bb2732b3 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240501 14:09:51.759613 7016 ts_tablet_manager.cc:1367] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:51.759956 7018 raft_consensus.cc:348] T ea69585d641342b8a1340339bb2732b3 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:51.760047 7018 raft_consensus.cc:374] T ea69585d641342b8a1340339bb2732b3 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.760076 7018 raft_consensus.cc:729] T ea69585d641342b8a1340339bb2732b3 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.760182 7018 consensus_queue.cc:260] T ea69585d641342b8a1340339bb2732b3 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:51.760466 7018 ts_tablet_manager.cc:1398] T ea69585d641342b8a1340339bb2732b3 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:51.761030 7016 raft_consensus.cc:348] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:51.761116 7016 raft_consensus.cc:374] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.761225 7016 raft_consensus.cc:729] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.761366 7016 consensus_queue.cc:260] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:51.761647 7016 ts_tablet_manager.cc:1398] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:51.774081 32554 raft_consensus.cc:1203] T f2bc06e76e2a41429d5625ebf99e5e5c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:51.782224 6832 raft_consensus.cc:482] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:51.782326 6832 raft_consensus.cc:504] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:51.782795 6832 leader_election.cc:288] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:51.783484 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6208e4b3d25941848b2ca533edd5a7b9" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:51.783624 30406 raft_consensus.cc:2454] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:51.783914 32618 leader_election.cc:302] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:51.784210 6832 raft_consensus.cc:2781] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:51.784297 6832 raft_consensus.cc:482] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:51.784338 6832 raft_consensus.cc:3037] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.784569 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6208e4b3d25941848b2ca533edd5a7b9" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:51.784678 348 raft_consensus.cc:2454] T 6208e4b3d25941848b2ca533edd5a7b9 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:51.785966 327 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:51.786942 6832 raft_consensus.cc:504] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:51.787279 6832 leader_election.cc:288] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Requested vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:51.788007 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6208e4b3d25941848b2ca533edd5a7b9" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:51.788136 30406 raft_consensus.cc:3037] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.788755 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6208e4b3d25941848b2ca533edd5a7b9" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:51.788856 348 raft_consensus.cc:3037] T 6208e4b3d25941848b2ca533edd5a7b9 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.789834 348 raft_consensus.cc:2454] T 6208e4b3d25941848b2ca533edd5a7b9 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:51.790138 32620 leader_election.cc:302] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:51.790290 6832 raft_consensus.cc:2781] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:51.790369 6832 raft_consensus.cc:686] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Becoming Leader. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
I20240501 14:09:51.790515 6832 consensus_queue.cc:237] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:51.791993 30406 raft_consensus.cc:2454] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:51.802039 29870 catalog_manager.cc:5484] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db935501e9104de5b15c8780681d6c38 (127.22.252.135). New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:51.802269 32665 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:51.804750 30255 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:51.805022 30126 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:51.805061 32401 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:51.806246 32533 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:51.817831 6929 raft_consensus.cc:482] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:51.817926 6929 raft_consensus.cc:504] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:51.818306 6929 leader_election.cc:288] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:51.818652 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce3093e24347f3a4e121af452c6361" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:51.818770 32415 raft_consensus.cc:2454] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:51.819002 32489 leader_election.cc:302] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:51.819394 6929 raft_consensus.cc:2781] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:51.819447 6929 raft_consensus.cc:482] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:51.819478 6929 raft_consensus.cc:3037] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.819959 5788 raft_consensus.cc:482] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:51.820050 5788 raft_consensus.cc:504] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:51.820528 5788 leader_election.cc:288] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:51.820539 6929 raft_consensus.cc:504] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:51.820837 6929 leader_election.cc:288] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:51.820883 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918ddead97f64b10ac165dada9078d41" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:51.821013 30277 raft_consensus.cc:2454] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:51.821177 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce3093e24347f3a4e121af452c6361" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:51.821278 32415 raft_consensus.cc:3037] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.821339 32752 leader_election.cc:302] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:51.821491 5788 raft_consensus.cc:2781] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:51.821529 5788 raft_consensus.cc:482] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:51.821561 5788 raft_consensus.cc:3037] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.822209 32415 raft_consensus.cc:2454] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:51.822523 5788 raft_consensus.cc:504] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:51.822561 32489 leader_election.cc:302] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:51.822837 5788 leader_election.cc:288] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:51.823196 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918ddead97f64b10ac165dada9078d41" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:51.823305 30277 raft_consensus.cc:3037] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.823357 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce3093e24347f3a4e121af452c6361" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:51.823377 30408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce3093e24347f3a4e121af452c6361" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:51.823489 30406 raft_consensus.cc:2454] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:51.824313 30277 raft_consensus.cc:2454] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:51.824705 32752 leader_election.cc:302] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:51.825045 30408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918ddead97f64b10ac165dada9078d41" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:51.825143 30408 raft_consensus.cc:2454] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:51.825013 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918ddead97f64b10ac165dada9078d41" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:51.825219 30406 raft_consensus.cc:3037] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.826110 6929 raft_consensus.cc:2781] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:51.826156 30406 raft_consensus.cc:2454] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:51.826198 6929 raft_consensus.cc:686] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:09:51.826323 6929 consensus_queue.cc:237] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:51.826730 5788 raft_consensus.cc:2781] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:51.826821 5788 raft_consensus.cc:686] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:51.826937 5788 consensus_queue.cc:237] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:51.829417 29867 catalog_manager.cc:5484] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:51.833040 29867 catalog_manager.cc:5484] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:51.843976 30388 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 14:09:51.849224 7040 log.cc:924] Time spent T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20240501 14:09:51.854351 29993 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:51.889102 5788 raft_consensus.cc:482] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:51.889221 5788 raft_consensus.cc:504] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:51.889660 5788 leader_election.cc:288] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:51.889946 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea69585d641342b8a1340339bb2732b3" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:51.890069 32554 raft_consensus.cc:2454] T ea69585d641342b8a1340339bb2732b3 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:51.890059 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea69585d641342b8a1340339bb2732b3" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:51.890219 32685 raft_consensus.cc:2454] T ea69585d641342b8a1340339bb2732b3 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:51.890348 32753 leader_election.cc:302] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:51.890506 5788 raft_consensus.cc:2781] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:51.890554 5788 raft_consensus.cc:482] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:51.890594 5788 raft_consensus.cc:3037] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.891580 5788 raft_consensus.cc:504] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:51.891897 5788 leader_election.cc:288] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:51.892365 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea69585d641342b8a1340339bb2732b3" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:51.892474 32554 raft_consensus.cc:3037] T ea69585d641342b8a1340339bb2732b3 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.892719 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea69585d641342b8a1340339bb2732b3" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:51.892822 32685 raft_consensus.cc:3037] T ea69585d641342b8a1340339bb2732b3 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.893469 32554 raft_consensus.cc:2454] T ea69585d641342b8a1340339bb2732b3 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:51.893729 32753 leader_election.cc:302] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:51.893851 32685 raft_consensus.cc:2454] T ea69585d641342b8a1340339bb2732b3 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:51.893910 5788 raft_consensus.cc:2781] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:51.894002 5788 raft_consensus.cc:686] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:51.894147 5788 consensus_queue.cc:237] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:51.896684 29867 catalog_manager.cc:5484] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:51.906599 30013 consensus_queue.cc:237] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:51.907816 32554 raft_consensus.cc:1261] T 7b3a90c03de64d6891dda267167d1d7b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:51.907816 30277 raft_consensus.cc:1261] T 7b3a90c03de64d6891dda267167d1d7b P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:51.908284 6075 consensus_queue.cc:1035] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:51.908478 6819 consensus_queue.cc:1035] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:51.910127 6075 raft_consensus.cc:2932] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:51.910964 32554 raft_consensus.cc:2932] T 7b3a90c03de64d6891dda267167d1d7b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:51.911738 30277 raft_consensus.cc:2932] T 7b3a90c03de64d6891dda267167d1d7b P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:51.913260 29867 catalog_manager.cc:5484] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e reported cstate change: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:51.917711 32401 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b3a90c03de64d6891dda267167d1d7b with delete_type TABLET_DATA_TOMBSTONED (TS a91029b90bf9415a89a2f70736759ad3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:51.918179 7066 tablet_replica.cc:330] T 7b3a90c03de64d6891dda267167d1d7b P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:51.918359 7066 raft_consensus.cc:2229] T 7b3a90c03de64d6891dda267167d1d7b P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:51.918504 7066 raft_consensus.cc:2258] T 7b3a90c03de64d6891dda267167d1d7b P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:51.918992 7066 ts_tablet_manager.cc:1875] T 7b3a90c03de64d6891dda267167d1d7b P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:51.921448 7066 ts_tablet_manager.cc:1888] T 7b3a90c03de64d6891dda267167d1d7b P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:09:51.921526 7066 log.cc:1196] T 7b3a90c03de64d6891dda267167d1d7b P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/7b3a90c03de64d6891dda267167d1d7b
I20240501 14:09:51.922050 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 7b3a90c03de64d6891dda267167d1d7b (table rebalancer_extra_table_10 [id=c8e44abe11554acda26e71b8a11e2e0e]) successfully deleted
I20240501 14:09:51.937685 29870 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:51.941358 30264 raft_consensus.cc:1261] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:51.941568 30406 raft_consensus.cc:1261] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:51.941726 5788 consensus_queue.cc:1035] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:51.942231 5788 consensus_queue.cc:1035] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:51.945348 29866 catalog_manager.cc:5576] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103) does not have the latest schema for tablet 918ddead97f64b10ac165dada9078d41 (table rebalancer_dynamic_table [id=460b17cdb5434a0faf4015b55014b0cc]). Expected version 1 got 0
I20240501 14:09:51.946791 29866 catalog_manager.cc:5576] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357) does not have the latest schema for tablet 918ddead97f64b10ac165dada9078d41 (table rebalancer_dynamic_table [id=460b17cdb5434a0faf4015b55014b0cc]). Expected version 1 got 0
I20240501 14:09:51.947714 6844 tablet.cc:1722] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:51.948014 6844 tablet.cc:1620] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:51.948184 7083 tablet.cc:1722] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:51.948508 7083 tablet.cc:1620] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7: MemRowSet was empty: no flush needed.
I20240501 14:09:51.951308 7084 tablet.cc:1722] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:51.951866 7084 tablet.cc:1620] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a: MemRowSet was empty: no flush needed.
I20240501 14:09:51.952739 29866 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=460b17cdb5434a0faf4015b55014b0cc] alter complete (version 1)
I20240501 14:09:52.049513 32533 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:52.052006 30126 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:52.054899 30388 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:52.054797 29866 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:52.055851 29993 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:52.057044 32401 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:52.058594 30255 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:52.061702 327 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:52.071641 6844 tablet.cc:1722] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:52.071983 6844 tablet.cc:1620] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:52.081499 7083 tablet.cc:1722] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:52.081822 7083 tablet.cc:1620] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7: MemRowSet was empty: no flush needed.
I20240501 14:09:52.082337 29866 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=460b17cdb5434a0faf4015b55014b0cc] alter complete (version 2)
I20240501 14:09:52.093806 7084 tablet.cc:1722] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:52.094101 7084 tablet.cc:1620] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a: MemRowSet was empty: no flush needed.
I20240501 14:09:52.104544 32665 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:52.148204 32553 consensus_queue.cc:237] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:52.149470 32685 raft_consensus.cc:1261] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:52.149598 30264 raft_consensus.cc:1261] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:52.149885 6456 consensus_queue.cc:1035] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:52.150039 6929 consensus_queue.cc:1035] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:52.152014 6929 raft_consensus.cc:2932] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:52.153548 32685 raft_consensus.cc:2932] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:52.155714 29866 catalog_manager.cc:5484] T 1e03e76474584273ae050595a2286559 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:52.159431 32401 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e03e76474584273ae050595a2286559 with delete_type TABLET_DATA_TOMBSTONED (TS a91029b90bf9415a89a2f70736759ad3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:52.159684 7066 tablet_replica.cc:330] T 1e03e76474584273ae050595a2286559 P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:52.159852 7066 raft_consensus.cc:2229] T 1e03e76474584273ae050595a2286559 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:52.159983 7066 raft_consensus.cc:2258] T 1e03e76474584273ae050595a2286559 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:52.160399 7066 ts_tablet_manager.cc:1875] T 1e03e76474584273ae050595a2286559 P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:52.162058 7066 ts_tablet_manager.cc:1888] T 1e03e76474584273ae050595a2286559 P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:09:52.162127 7066 log.cc:1196] T 1e03e76474584273ae050595a2286559 P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/1e03e76474584273ae050595a2286559
I20240501 14:09:52.162608 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 1e03e76474584273ae050595a2286559 (table rebalancer_extra_table_12 [id=dcad19a5572a46a992d9c7fd61cfcf2b]) successfully deleted
I20240501 14:09:52.167421 30264 raft_consensus.cc:2932] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:52.190936 29873 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:52.196079 6844 tablet.cc:1722] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:52.197067 7084 tablet.cc:1722] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:52.197100 7083 tablet.cc:1722] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:52.202363 29873 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=460b17cdb5434a0faf4015b55014b0cc] alter complete (version 3)
I20240501 14:09:52.243367 32553 consensus_queue.cc:237] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:52.244837 348 raft_consensus.cc:1261] T 21efaac3a3d643679c5abaaddb9e5a3a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:52.245188 32685 raft_consensus.cc:1261] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:52.245303 32489 consensus_peers.cc:485] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 -> Peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): Couldn't send request to peer 1d0bb69611af4876a2b3b33c4766e13a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21efaac3a3d643679c5abaaddb9e5a3a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:52.245519 6456 consensus_queue.cc:1035] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:52.245728 7186 consensus_queue.cc:1035] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:52.247510 6456 raft_consensus.cc:2932] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:52.247916 348 raft_consensus.cc:2932] T 21efaac3a3d643679c5abaaddb9e5a3a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:52.252079 29865 catalog_manager.cc:5484] T 21efaac3a3d643679c5abaaddb9e5a3a P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:52.252385 32685 raft_consensus.cc:2932] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:52.269892 30146 consensus_queue.cc:237] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:52.271296 30406 raft_consensus.cc:1261] T f740d894942a4bb584ca5688ef823d55 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:52.271518 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f740d894942a4bb584ca5688ef823d55. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:52.271523 30264 raft_consensus.cc:1261] T f740d894942a4bb584ca5688ef823d55 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:52.271744 7205 consensus_queue.cc:1035] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:52.271899 6580 consensus_queue.cc:1035] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:52.273854 6580 raft_consensus.cc:2932] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:52.274278 30264 raft_consensus.cc:2932] T f740d894942a4bb584ca5688ef823d55 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:52.277161 29873 catalog_manager.cc:5484] T f740d894942a4bb584ca5688ef823d55 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:52.277944 30406 raft_consensus.cc:2932] T f740d894942a4bb584ca5688ef823d55 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:52.287083 5788 consensus_queue.cc:1035] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:52.291796 5788 consensus_queue.cc:1035] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:52.303661 29873 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:52.303793 29873 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:52.304091 6832 consensus_queue.cc:1035] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:52.305871 29873 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 918ddead97f64b10ac165dada9078d41
I20240501 14:09:52.306473 327 tablet_service.cc:1514] Processing DeleteTablet for tablet 918ddead97f64b10ac165dada9078d41 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:52 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:52.306802 30388 tablet_service.cc:1514] Processing DeleteTablet for tablet 918ddead97f64b10ac165dada9078d41 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:52 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:52.307677 30255 tablet_service.cc:1514] Processing DeleteTablet for tablet 918ddead97f64b10ac165dada9078d41 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:52 UTC) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:52.308099 6858 consensus_queue.cc:1035] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:52.308218 7226 tablet_replica.cc:330] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:52.308393 7226 raft_consensus.cc:2229] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:52.308518 7226 raft_consensus.cc:2258] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:52.309127 7226 ts_tablet_manager.cc:1875] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:52.309912 7227 tablet_replica.cc:330] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:52.310067 7227 raft_consensus.cc:2229] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:52.310189 7227 raft_consensus.cc:2258] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:52.310408 7225 tablet_replica.cc:330] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:52.310580 7225 raft_consensus.cc:2229] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:52.310801 7225 raft_consensus.cc:2258] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:52.311002 7227 ts_tablet_manager.cc:1875] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:52.313248 30408 raft_consensus.cc:3037] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.315584 7225 ts_tablet_manager.cc:1875] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:52.316460 6929 consensus_queue.cc:1035] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:52.318754 7225 ts_tablet_manager.cc:1888] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:52.318850 7225 log.cc:1196] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/918ddead97f64b10ac165dada9078d41
I20240501 14:09:52.319155 7225 ts_tablet_manager.cc:1909] T 918ddead97f64b10ac165dada9078d41 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:09:52.319809 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 918ddead97f64b10ac165dada9078d41 (table rebalancer_dynamic_table_new_name [id=460b17cdb5434a0faf4015b55014b0cc]) successfully deleted
I20240501 14:09:52.320529 7226 ts_tablet_manager.cc:1888] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:52.320649 7226 log.cc:1196] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/918ddead97f64b10ac165dada9078d41
I20240501 14:09:52.320837 7227 ts_tablet_manager.cc:1888] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:52.320914 7227 log.cc:1196] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/918ddead97f64b10ac165dada9078d41
I20240501 14:09:52.321082 7226 ts_tablet_manager.cc:1909] T 918ddead97f64b10ac165dada9078d41 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:52.321456 7227 ts_tablet_manager.cc:1909] T 918ddead97f64b10ac165dada9078d41 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting consensus metadata
I20240501 14:09:52.321873 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 918ddead97f64b10ac165dada9078d41 (table rebalancer_dynamic_table_new_name [id=460b17cdb5434a0faf4015b55014b0cc]) successfully deleted
I20240501 14:09:52.322088 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet 918ddead97f64b10ac165dada9078d41 (table rebalancer_dynamic_table_new_name [id=460b17cdb5434a0faf4015b55014b0cc]) successfully deleted
I20240501 14:09:52.324375 6929 consensus_queue.cc:1035] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:52.394469 29993 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:52.394515 30255 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:52.398149 32533 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:52.400941 32401 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:52.402246 30388 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:52.408897 29873 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:52.418715 29868 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:52.428205 327 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:52.431035 30126 tablet_service.cc:1467] Processing CreateTablet for tablet 3758e4f9cfa145db8438d6be1a7aca0f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e43424dadcf24e538efa4828a56d7035]), partition=
I20240501 14:09:52.431277 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3758e4f9cfa145db8438d6be1a7aca0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.433462 30255 tablet_service.cc:1467] Processing CreateTablet for tablet 3758e4f9cfa145db8438d6be1a7aca0f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e43424dadcf24e538efa4828a56d7035]), partition=
I20240501 14:09:52.433701 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3758e4f9cfa145db8438d6be1a7aca0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.435155 32401 tablet_service.cc:1467] Processing CreateTablet for tablet 3758e4f9cfa145db8438d6be1a7aca0f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e43424dadcf24e538efa4828a56d7035]), partition=
I20240501 14:09:52.435351 32401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3758e4f9cfa145db8438d6be1a7aca0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.435508 30126 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:52.438771 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc60a3f814548f98b09bd7758dc3b75 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=9ad5e8a877804817a0b33b99b908abad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:52.439044 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc60a3f814548f98b09bd7758dc3b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.442401 7275 tablet_bootstrap.cc:492] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:52.442603 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 9030fb3b12cd40c493bb5f30bac37090 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=9ad5e8a877804817a0b33b99b908abad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:52.442883 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9030fb3b12cd40c493bb5f30bac37090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.443348 7275 tablet_bootstrap.cc:654] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.444046 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 4e84e8237a0b4809a8bf19bf242c78b4 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=9ad5e8a877804817a0b33b99b908abad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:52.444303 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e84e8237a0b4809a8bf19bf242c78b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.444612 29995 tablet_service.cc:1467] Processing CreateTablet for tablet 9030fb3b12cd40c493bb5f30bac37090 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=9ad5e8a877804817a0b33b99b908abad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:52.444640 327 tablet_service.cc:1467] Processing CreateTablet for tablet 4e84e8237a0b4809a8bf19bf242c78b4 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=9ad5e8a877804817a0b33b99b908abad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:52.444802 29995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9030fb3b12cd40c493bb5f30bac37090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.444871 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e84e8237a0b4809a8bf19bf242c78b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.446847 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 4e84e8237a0b4809a8bf19bf242c78b4 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=9ad5e8a877804817a0b33b99b908abad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:52.447091 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e84e8237a0b4809a8bf19bf242c78b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.441882 30126 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc60a3f814548f98b09bd7758dc3b75 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=9ad5e8a877804817a0b33b99b908abad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:52.448827 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc60a3f814548f98b09bd7758dc3b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.450474 32401 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc60a3f814548f98b09bd7758dc3b75 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=9ad5e8a877804817a0b33b99b908abad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:52.450673 32401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc60a3f814548f98b09bd7758dc3b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.452975 32533 tablet_service.cc:1467] Processing CreateTablet for tablet 9030fb3b12cd40c493bb5f30bac37090 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=9ad5e8a877804817a0b33b99b908abad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:52.453233 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9030fb3b12cd40c493bb5f30bac37090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.455891 32665 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:52.456153 7279 tablet_bootstrap.cc:492] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:52.458432 7279 tablet_bootstrap.cc:654] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.468659 7274 tablet_bootstrap.cc:492] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:52.468899 7281 tablet_bootstrap.cc:492] T 9030fb3b12cd40c493bb5f30bac37090 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:52.471555 7280 tablet_bootstrap.cc:492] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:52.472272 7281 tablet_bootstrap.cc:654] T 9030fb3b12cd40c493bb5f30bac37090 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.472484 7274 tablet_bootstrap.cc:654] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.473111 7277 tablet_bootstrap.cc:492] T 9fc60a3f814548f98b09bd7758dc3b75 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:52.473235 7276 tablet_bootstrap.cc:492] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:52.474428 7276 tablet_bootstrap.cc:654] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.475226 7277 tablet_bootstrap.cc:654] T 9fc60a3f814548f98b09bd7758dc3b75 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.475543 7280 tablet_bootstrap.cc:654] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.475570 7274 tablet_bootstrap.cc:492] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:52.475700 7274 ts_tablet_manager.cc:1367] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:52.476151 7274 raft_consensus.cc:348] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:52.476332 7274 raft_consensus.cc:374] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.476382 7274 raft_consensus.cc:729] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.476526 7274 consensus_queue.cc:260] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:52.477129 7281 tablet_bootstrap.cc:492] T 9030fb3b12cd40c493bb5f30bac37090 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:52.477267 7281 ts_tablet_manager.cc:1367] T 9030fb3b12cd40c493bb5f30bac37090 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240501 14:09:52.477535 7274 ts_tablet_manager.cc:1398] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:52.477766 7274 tablet_bootstrap.cc:492] T 9fc60a3f814548f98b09bd7758dc3b75 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:52.479180 7274 tablet_bootstrap.cc:654] T 9fc60a3f814548f98b09bd7758dc3b75 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.477708 7281 raft_consensus.cc:348] T 9030fb3b12cd40c493bb5f30bac37090 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:52.479380 7281 raft_consensus.cc:374] T 9030fb3b12cd40c493bb5f30bac37090 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.479480 7281 raft_consensus.cc:729] T 9030fb3b12cd40c493bb5f30bac37090 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.479661 7281 consensus_queue.cc:260] T 9030fb3b12cd40c493bb5f30bac37090 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:52.480437 7276 tablet_bootstrap.cc:492] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:52.480520 7276 ts_tablet_manager.cc:1367] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:52.480904 7276 raft_consensus.cc:348] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:52.480989 7276 raft_consensus.cc:374] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.481021 7276 raft_consensus.cc:729] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.481169 7276 consensus_queue.cc:260] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:52.481494 7276 ts_tablet_manager.cc:1398] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:52.481719 7276 tablet_bootstrap.cc:492] T 4e84e8237a0b4809a8bf19bf242c78b4 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:52.482700 7276 tablet_bootstrap.cc:654] T 4e84e8237a0b4809a8bf19bf242c78b4 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.481545 7281 ts_tablet_manager.cc:1398] T 9030fb3b12cd40c493bb5f30bac37090 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:09:52.484092 7277 tablet_bootstrap.cc:492] T 9fc60a3f814548f98b09bd7758dc3b75 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:52.484222 7277 ts_tablet_manager.cc:1367] T 9fc60a3f814548f98b09bd7758dc3b75 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:09:52.484858 7279 tablet_bootstrap.cc:492] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:52.485013 7279 ts_tablet_manager.cc:1367] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.002s
I20240501 14:09:52.485360 7276 tablet_bootstrap.cc:492] T 4e84e8237a0b4809a8bf19bf242c78b4 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:52.485339 7277 raft_consensus.cc:348] T 9fc60a3f814548f98b09bd7758dc3b75 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:52.485432 7277 raft_consensus.cc:374] T 9fc60a3f814548f98b09bd7758dc3b75 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.485451 7279 raft_consensus.cc:348] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:52.485538 7276 ts_tablet_manager.cc:1367] T 4e84e8237a0b4809a8bf19bf242c78b4 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:52.485580 7279 raft_consensus.cc:374] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.485595 7277 raft_consensus.cc:729] T 9fc60a3f814548f98b09bd7758dc3b75 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.485668 7279 raft_consensus.cc:729] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.485194 7275 tablet_bootstrap.cc:492] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:52.485780 7277 consensus_queue.cc:260] T 9fc60a3f814548f98b09bd7758dc3b75 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:52.486032 7276 raft_consensus.cc:348] T 4e84e8237a0b4809a8bf19bf242c78b4 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:52.486064 7279 consensus_queue.cc:260] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:52.486160 6424 raft_consensus.cc:482] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:52.486217 6424 raft_consensus.cc:504] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:52.486193 7276 raft_consensus.cc:374] T 4e84e8237a0b4809a8bf19bf242c78b4 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.486367 7276 raft_consensus.cc:729] T 4e84e8237a0b4809a8bf19bf242c78b4 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.486636 6424 leader_election.cc:288] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:52.487116 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e84e8237a0b4809a8bf19bf242c78b4" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:52.487156 7275 ts_tablet_manager.cc:1367] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.000s
I20240501 14:09:52.487386 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e84e8237a0b4809a8bf19bf242c78b4" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:52.487658 7275 raft_consensus.cc:348] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:52.487766 7275 raft_consensus.cc:374] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.487802 7275 raft_consensus.cc:729] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
W20240501 14:09:52.487782 32752 leader_election.cc:341] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:52.486536 7276 consensus_queue.cc:260] T 4e84e8237a0b4809a8bf19bf242c78b4 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:52.488138 7279 ts_tablet_manager.cc:1398] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:52.488714 7276 ts_tablet_manager.cc:1398] T 4e84e8237a0b4809a8bf19bf242c78b4 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:52.488915 7277 ts_tablet_manager.cc:1398] T 9fc60a3f814548f98b09bd7758dc3b75 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:52.489084 32416 raft_consensus.cc:2454] T 4e84e8237a0b4809a8bf19bf242c78b4 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:52.489518 32752 leader_election.cc:302] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters: 93f2518fbadf4f9991cda93afacb363e
I20240501 14:09:52.489715 6424 raft_consensus.cc:2781] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:52.489760 6424 raft_consensus.cc:482] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:52.489794 6424 raft_consensus.cc:3037] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.490793 7277 tablet_bootstrap.cc:492] T 9030fb3b12cd40c493bb5f30bac37090 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:52.490793 6424 raft_consensus.cc:504] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:52.491089 6424 leader_election.cc:288] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:52.491252 7276 tablet_bootstrap.cc:492] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:52.491772 7277 tablet_bootstrap.cc:654] T 9030fb3b12cd40c493bb5f30bac37090 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.491837 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e84e8237a0b4809a8bf19bf242c78b4" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:52.492074 7276 tablet_bootstrap.cc:654] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.492062 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e84e8237a0b4809a8bf19bf242c78b4" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:52.492165 32416 raft_consensus.cc:3037] T 4e84e8237a0b4809a8bf19bf242c78b4 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.487911 7275 consensus_queue.cc:260] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:52.493132 32416 raft_consensus.cc:2454] T 4e84e8237a0b4809a8bf19bf242c78b4 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:52.493362 7275 ts_tablet_manager.cc:1398] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:52.494786 32752 leader_election.cc:302] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:52.495079 7082 raft_consensus.cc:2781] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:52.495144 7082 raft_consensus.cc:686] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:52.495249 7082 consensus_queue.cc:237] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
W20240501 14:09:52.496647 32752 leader_election.cc:341] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:52.497643 7280 tablet_bootstrap.cc:492] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:52.498256 7276 tablet_bootstrap.cc:492] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:52.498353 7276 ts_tablet_manager.cc:1367] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240501 14:09:52.498576 7274 tablet_bootstrap.cc:492] T 9fc60a3f814548f98b09bd7758dc3b75 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:52.498654 7274 ts_tablet_manager.cc:1367] T 9fc60a3f814548f98b09bd7758dc3b75 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240501 14:09:52.498839 7276 raft_consensus.cc:348] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:52.498920 7276 raft_consensus.cc:374] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.498947 7276 raft_consensus.cc:729] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.499043 7276 consensus_queue.cc:260] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:52.499138 7274 raft_consensus.cc:348] T 9fc60a3f814548f98b09bd7758dc3b75 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:52.499236 7274 raft_consensus.cc:374] T 9fc60a3f814548f98b09bd7758dc3b75 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.499270 7274 raft_consensus.cc:729] T 9fc60a3f814548f98b09bd7758dc3b75 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.499281 7276 ts_tablet_manager.cc:1398] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:52.499382 7274 consensus_queue.cc:260] T 9fc60a3f814548f98b09bd7758dc3b75 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:52.499697 7274 ts_tablet_manager.cc:1398] T 9fc60a3f814548f98b09bd7758dc3b75 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:52.500558 29866 catalog_manager.cc:5484] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:52.501236 7280 ts_tablet_manager.cc:1367] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20240501 14:09:52.501754 7280 raft_consensus.cc:348] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:52.501931 7280 raft_consensus.cc:374] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.501974 7280 raft_consensus.cc:729] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.502102 7280 consensus_queue.cc:260] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:52.502447 7280 ts_tablet_manager.cc:1398] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:52.504526 7277 tablet_bootstrap.cc:492] T 9030fb3b12cd40c493bb5f30bac37090 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:52.504601 7277 ts_tablet_manager.cc:1367] T 9030fb3b12cd40c493bb5f30bac37090 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20240501 14:09:52.505002 7277 raft_consensus.cc:348] T 9030fb3b12cd40c493bb5f30bac37090 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:52.505091 7277 raft_consensus.cc:374] T 9030fb3b12cd40c493bb5f30bac37090 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.505120 7277 raft_consensus.cc:729] T 9030fb3b12cd40c493bb5f30bac37090 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.505223 7277 consensus_queue.cc:260] T 9030fb3b12cd40c493bb5f30bac37090 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:52.505465 7277 ts_tablet_manager.cc:1398] T 9030fb3b12cd40c493bb5f30bac37090 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:52.505602 7277 tablet_bootstrap.cc:492] T 4e84e8237a0b4809a8bf19bf242c78b4 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:52.506418 7277 tablet_bootstrap.cc:654] T 4e84e8237a0b4809a8bf19bf242c78b4 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.510061 7277 tablet_bootstrap.cc:492] T 4e84e8237a0b4809a8bf19bf242c78b4 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:52.510149 7277 ts_tablet_manager.cc:1367] T 4e84e8237a0b4809a8bf19bf242c78b4 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:09:52.510541 7277 raft_consensus.cc:348] T 4e84e8237a0b4809a8bf19bf242c78b4 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:52.510622 7277 raft_consensus.cc:374] T 4e84e8237a0b4809a8bf19bf242c78b4 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.510648 7277 raft_consensus.cc:729] T 4e84e8237a0b4809a8bf19bf242c78b4 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.510758 7277 consensus_queue.cc:260] T 4e84e8237a0b4809a8bf19bf242c78b4 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:52.511001 7277 ts_tablet_manager.cc:1398] T 4e84e8237a0b4809a8bf19bf242c78b4 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:52.525255 6905 raft_consensus.cc:482] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:52.525357 6905 raft_consensus.cc:504] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:52.525791 6905 leader_election.cc:288] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:52.526094 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc60a3f814548f98b09bd7758dc3b75" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:52.526124 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc60a3f814548f98b09bd7758dc3b75" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:52.526211 30013 raft_consensus.cc:2454] T 9fc60a3f814548f98b09bd7758dc3b75 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:52.526243 30146 raft_consensus.cc:2454] T 9fc60a3f814548f98b09bd7758dc3b75 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:52.526501 32359 leader_election.cc:302] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:52.526691 6905 raft_consensus.cc:2781] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:52.526749 6905 raft_consensus.cc:482] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:52.526798 6905 raft_consensus.cc:3037] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.527880 6905 raft_consensus.cc:504] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:52.528280 6905 leader_election.cc:288] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:52.528550 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc60a3f814548f98b09bd7758dc3b75" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:52.528607 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc60a3f814548f98b09bd7758dc3b75" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:52.528678 30013 raft_consensus.cc:3037] T 9fc60a3f814548f98b09bd7758dc3b75 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.528713 30146 raft_consensus.cc:3037] T 9fc60a3f814548f98b09bd7758dc3b75 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.529629 30013 raft_consensus.cc:2454] T 9fc60a3f814548f98b09bd7758dc3b75 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:52.529629 30146 raft_consensus.cc:2454] T 9fc60a3f814548f98b09bd7758dc3b75 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:52.529897 32359 leader_election.cc:302] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:52.530125 6905 raft_consensus.cc:2781] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:52.530304 6905 raft_consensus.cc:686] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:52.530439 6905 consensus_queue.cc:237] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:52.532665 29873 catalog_manager.cc:5484] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:52.655965 6028 raft_consensus.cc:482] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:52.656083 6028 raft_consensus.cc:504] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:52.656549 6028 leader_election.cc:288] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:52.656852 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9030fb3b12cd40c493bb5f30bac37090" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:52.656900 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9030fb3b12cd40c493bb5f30bac37090" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:52.656983 32553 raft_consensus.cc:2454] T 9030fb3b12cd40c493bb5f30bac37090 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:52.657052 30013 raft_consensus.cc:2454] T 9030fb3b12cd40c493bb5f30bac37090 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:52.657269 30344 leader_election.cc:302] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:52.657502 6028 raft_consensus.cc:2781] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:52.657552 6028 raft_consensus.cc:482] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:52.657608 6028 raft_consensus.cc:3037] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.658844 6028 raft_consensus.cc:504] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:52.659200 6028 leader_election.cc:288] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:52.659518 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9030fb3b12cd40c493bb5f30bac37090" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:52.659632 32553 raft_consensus.cc:3037] T 9030fb3b12cd40c493bb5f30bac37090 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.659639 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9030fb3b12cd40c493bb5f30bac37090" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:52.659746 30013 raft_consensus.cc:3037] T 9030fb3b12cd40c493bb5f30bac37090 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.660730 32553 raft_consensus.cc:2454] T 9030fb3b12cd40c493bb5f30bac37090 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:52.661041 30013 raft_consensus.cc:2454] T 9030fb3b12cd40c493bb5f30bac37090 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:52.661154 30344 leader_election.cc:302] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:52.661352 6028 raft_consensus.cc:2781] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:52.661442 6028 raft_consensus.cc:686] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:52.661582 6028 consensus_queue.cc:237] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:52.663997 29873 catalog_manager.cc:5484] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:52.679926 7338 ts_tablet_manager.cc:902] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a: Initiating tablet copy from peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:52.680574 7338 tablet_copy_client.cc:308] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.134:33497
I20240501 14:09:52.681049 32572 tablet_copy_service.cc:140] P 5f8a0cfdfdca4148a75686dc88059cf3: Received BeginTabletCopySession request for tablet 21efaac3a3d643679c5abaaddb9e5a3a from peer 1d0bb69611af4876a2b3b33c4766e13a ({username='slave'} at 127.22.252.131:56783)
I20240501 14:09:52.681170 32572 tablet_copy_service.cc:161] P 5f8a0cfdfdca4148a75686dc88059cf3: Beginning new tablet copy session on tablet 21efaac3a3d643679c5abaaddb9e5a3a from peer 1d0bb69611af4876a2b3b33c4766e13a at {username='slave'} at 127.22.252.131:56783: session id = 1d0bb69611af4876a2b3b33c4766e13a-21efaac3a3d643679c5abaaddb9e5a3a
I20240501 14:09:52.682317 32572 tablet_copy_source_session.cc:205] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:52.683838 7338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21efaac3a3d643679c5abaaddb9e5a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.685930 7338 tablet_copy_client.cc:783] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:52.686118 7338 tablet_copy_client.cc:647] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:52.687209 7338 tablet_copy_client.cc:523] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:52.688297 7338 tablet_bootstrap.cc:492] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:52.691602 7338 tablet_bootstrap.cc:492] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:52.691830 7338 tablet_bootstrap.cc:492] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap complete.
I20240501 14:09:52.691973 7338 ts_tablet_manager.cc:1367] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240501 14:09:52.692416 7338 raft_consensus.cc:348] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:52.692548 7338 raft_consensus.cc:729] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.692724 7338 consensus_queue.cc:260] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:52.693009 7338 ts_tablet_manager.cc:1398] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:52.693542 32572 tablet_copy_service.cc:342] P 5f8a0cfdfdca4148a75686dc88059cf3: Request end of tablet copy session 1d0bb69611af4876a2b3b33c4766e13a-21efaac3a3d643679c5abaaddb9e5a3a received from {username='slave'} at 127.22.252.131:56783
I20240501 14:09:52.693627 32572 tablet_copy_service.cc:433] P 5f8a0cfdfdca4148a75686dc88059cf3: ending tablet copy session 1d0bb69611af4876a2b3b33c4766e13a-21efaac3a3d643679c5abaaddb9e5a3a on tablet 21efaac3a3d643679c5abaaddb9e5a3a with peer 1d0bb69611af4876a2b3b33c4766e13a
I20240501 14:09:52.777500 6593 raft_consensus.cc:482] T 4e84e8237a0b4809a8bf19bf242c78b4 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:52.777645 6593 raft_consensus.cc:504] T 4e84e8237a0b4809a8bf19bf242c78b4 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:52.778076 6593 leader_election.cc:288] T 4e84e8237a0b4809a8bf19bf242c78b4 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:52.778499 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e84e8237a0b4809a8bf19bf242c78b4" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:52.778662 32416 raft_consensus.cc:2379] T 4e84e8237a0b4809a8bf19bf242c78b4 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93f2518fbadf4f9991cda93afacb363e in current term 1: Already voted for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in this term.
I20240501 14:09:52.778609 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e84e8237a0b4809a8bf19bf242c78b4" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:52.779021 29950 leader_election.cc:302] T 4e84e8237a0b4809a8bf19bf242c78b4 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3
I20240501 14:09:52.779181 6593 raft_consensus.cc:3037] T 4e84e8237a0b4809a8bf19bf242c78b4 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.783341 6593 raft_consensus.cc:2726] T 4e84e8237a0b4809a8bf19bf242c78b4 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:09:52.823201 7344 ts_tablet_manager.cc:902] T f740d894942a4bb584ca5688ef823d55 P db935501e9104de5b15c8780681d6c38: Initiating tablet copy from peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:52.823886 7344 tablet_copy_client.cc:308] T f740d894942a4bb584ca5688ef823d55 P db935501e9104de5b15c8780681d6c38: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.130:36457
I20240501 14:09:52.824481 30165 tablet_copy_service.cc:140] P d35c16dcb506426993811bb2519d7db3: Received BeginTabletCopySession request for tablet f740d894942a4bb584ca5688ef823d55 from peer db935501e9104de5b15c8780681d6c38 ({username='slave'} at 127.22.252.135:49787)
I20240501 14:09:52.824587 30165 tablet_copy_service.cc:161] P d35c16dcb506426993811bb2519d7db3: Beginning new tablet copy session on tablet f740d894942a4bb584ca5688ef823d55 from peer db935501e9104de5b15c8780681d6c38 at {username='slave'} at 127.22.252.135:49787: session id = db935501e9104de5b15c8780681d6c38-f740d894942a4bb584ca5688ef823d55
I20240501 14:09:52.825416 30165 tablet_copy_source_session.cc:205] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:52.826493 7344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f740d894942a4bb584ca5688ef823d55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.828753 7344 tablet_copy_client.cc:783] T f740d894942a4bb584ca5688ef823d55 P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:52.828918 7344 tablet_copy_client.cc:647] T f740d894942a4bb584ca5688ef823d55 P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:52.830271 7344 tablet_copy_client.cc:523] T f740d894942a4bb584ca5688ef823d55 P db935501e9104de5b15c8780681d6c38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:52.831399 7344 tablet_bootstrap.cc:492] T f740d894942a4bb584ca5688ef823d55 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:52.843868 6683 raft_consensus.cc:482] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:52.843986 6683 raft_consensus.cc:504] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:52.844425 6683 leader_election.cc:288] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:52.844738 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3758e4f9cfa145db8438d6be1a7aca0f" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:52.844784 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3758e4f9cfa145db8438d6be1a7aca0f" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:52.844866 30146 raft_consensus.cc:2454] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:52.844910 32416 raft_consensus.cc:2454] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:52.845192 30213 leader_election.cc:302] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:52.845497 6683 raft_consensus.cc:2781] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:52.845571 6683 raft_consensus.cc:482] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:52.845614 6683 raft_consensus.cc:3037] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.846647 6683 raft_consensus.cc:504] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:52.846962 6683 leader_election.cc:288] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:52.847456 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3758e4f9cfa145db8438d6be1a7aca0f" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:52.847563 30146 raft_consensus.cc:3037] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.848654 30146 raft_consensus.cc:2454] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:52.849140 30213 leader_election.cc:302] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:52.849910 6683 raft_consensus.cc:2781] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:52.849958 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3758e4f9cfa145db8438d6be1a7aca0f" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:52.850006 6683 raft_consensus.cc:686] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:52.850077 32416 raft_consensus.cc:3037] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.850425 6683 consensus_queue.cc:237] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:52.851095 32416 raft_consensus.cc:2454] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:52.852782 29873 catalog_manager.cc:5484] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:52.853386 7344 tablet_bootstrap.cc:492] T f740d894942a4bb584ca5688ef823d55 P db935501e9104de5b15c8780681d6c38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:52.853724 7344 tablet_bootstrap.cc:492] T f740d894942a4bb584ca5688ef823d55 P db935501e9104de5b15c8780681d6c38: Bootstrap complete.
I20240501 14:09:52.853905 7344 ts_tablet_manager.cc:1367] T f740d894942a4bb584ca5688ef823d55 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.004s
I20240501 14:09:52.854386 7344 raft_consensus.cc:348] T f740d894942a4bb584ca5688ef823d55 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:52.854502 7344 raft_consensus.cc:729] T f740d894942a4bb584ca5688ef823d55 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.854635 7344 consensus_queue.cc:260] T f740d894942a4bb584ca5688ef823d55 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:52.855046 7344 ts_tablet_manager.cc:1398] T f740d894942a4bb584ca5688ef823d55 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:52.855829 30165 tablet_copy_service.cc:342] P d35c16dcb506426993811bb2519d7db3: Request end of tablet copy session db935501e9104de5b15c8780681d6c38-f740d894942a4bb584ca5688ef823d55 received from {username='slave'} at 127.22.252.135:49787
I20240501 14:09:52.855917 30165 tablet_copy_service.cc:433] P d35c16dcb506426993811bb2519d7db3: ending tablet copy session db935501e9104de5b15c8780681d6c38-f740d894942a4bb584ca5688ef823d55 on tablet f740d894942a4bb584ca5688ef823d55 with peer db935501e9104de5b15c8780681d6c38
I20240501 14:09:52.886976 29993 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:52.889605 32533 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:52.894831 30126 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:52.895310 30388 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:52.896698 30255 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:52.899067 327 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:52.908095 32665 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:52.943959 32401 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:52.982450 6905 consensus_queue.cc:1035] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:52.986680 7305 consensus_queue.cc:1035] T 9fc60a3f814548f98b09bd7758dc3b75 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:52.999190 5788 consensus_queue.cc:1035] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:53.007015 5788 consensus_queue.cc:1035] T 4e84e8237a0b4809a8bf19bf242c78b4 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:53.024333 29865 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:53.029783 32416 raft_consensus.cc:1261] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:53.029843 30146 raft_consensus.cc:1261] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:53.030225 6683 consensus_queue.cc:1035] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:53.030382 6686 consensus_queue.cc:1035] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:53.033509 29865 catalog_manager.cc:5576] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457) does not have the latest schema for tablet 3758e4f9cfa145db8438d6be1a7aca0f (table rebalancer_dynamic_table [id=e43424dadcf24e538efa4828a56d7035]). Expected version 1 got 0
I20240501 14:09:53.034415 29872 catalog_manager.cc:5576] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265) does not have the latest schema for tablet 3758e4f9cfa145db8438d6be1a7aca0f (table rebalancer_dynamic_table [id=e43424dadcf24e538efa4828a56d7035]). Expected version 1 got 0
I20240501 14:09:53.035907 7387 tablet.cc:1722] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:53.036541 7387 tablet.cc:1620] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a: MemRowSet was empty: no flush needed.
I20240501 14:09:53.039669 29873 catalog_manager.cc:5576] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457) does not have the latest schema for tablet 3758e4f9cfa145db8438d6be1a7aca0f (table rebalancer_dynamic_table [id=e43424dadcf24e538efa4828a56d7035]). Expected version 1 got 0
I20240501 14:09:53.040270 7374 mvcc.cc:205] Tried to move back new op lower bound from 7022889341046734848 to 7022889340317040640. Current Snapshot: MvccSnapshot[applied={T|T < 7022889341046734848}]
I20240501 14:09:53.041373 7394 tablet.cc:1722] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:53.041821 7394 tablet.cc:1620] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:53.044178 29873 catalog_manager.cc:5576] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265) does not have the latest schema for tablet 3758e4f9cfa145db8438d6be1a7aca0f (table rebalancer_dynamic_table [id=e43424dadcf24e538efa4828a56d7035]). Expected version 1 got 0
I20240501 14:09:53.047545 7392 tablet.cc:1722] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:53.047921 7392 tablet.cc:1620] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3: MemRowSet was empty: no flush needed.
I20240501 14:09:53.050261 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=e43424dadcf24e538efa4828a56d7035] alter complete (version 1)
I20240501 14:09:53.077626 6185 consensus_queue.cc:1035] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:53.080950 6185 consensus_queue.cc:1035] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:53.134359 30388 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:53.139209 29993 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:53.140940 327 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:53.152570 30255 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:53.154799 29865 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:53.157092 30126 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:53.161247 32533 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:53.161621 7387 tablet.cc:1722] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:53.161737 32401 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:53.161911 7387 tablet.cc:1620] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a: MemRowSet was empty: no flush needed.
I20240501 14:09:53.169777 7394 tablet.cc:1722] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:53.170099 7394 tablet.cc:1620] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:53.176061 32665 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:53.176268 7392 tablet.cc:1722] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:53.176574 7392 tablet.cc:1620] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3: MemRowSet was empty: no flush needed.
I20240501 14:09:53.181030 29867 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:53.186064 29865 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=e43424dadcf24e538efa4828a56d7035] alter complete (version 2)
I20240501 14:09:53.198455 30255 tablet_service.cc:1467] Processing CreateTablet for tablet 42636717a44e4e3782037c48d8913307 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=eaceddc0ae4d496fa3ffc653930395b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:53.198508 30126 tablet_service.cc:1467] Processing CreateTablet for tablet 85cc7734181b4c89b7d50f9e81a2f583 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=eaceddc0ae4d496fa3ffc653930395b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:53.198752 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42636717a44e4e3782037c48d8913307. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.198767 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85cc7734181b4c89b7d50f9e81a2f583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.198721 30257 tablet_service.cc:1467] Processing CreateTablet for tablet bcec6adcdf4e471b86954e6d99528a22 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=eaceddc0ae4d496fa3ffc653930395b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:53.198951 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcec6adcdf4e471b86954e6d99528a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.201345 32401 tablet_service.cc:1467] Processing CreateTablet for tablet 42636717a44e4e3782037c48d8913307 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=eaceddc0ae4d496fa3ffc653930395b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:53.201586 32401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42636717a44e4e3782037c48d8913307. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.201735 30126 tablet_service.cc:1467] Processing CreateTablet for tablet bcec6adcdf4e471b86954e6d99528a22 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=eaceddc0ae4d496fa3ffc653930395b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:53.201927 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcec6adcdf4e471b86954e6d99528a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.204011 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 42636717a44e4e3782037c48d8913307 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=eaceddc0ae4d496fa3ffc653930395b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:53.204270 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42636717a44e4e3782037c48d8913307. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.204463 7439 tablet_bootstrap.cc:492] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:53.205338 7439 tablet_bootstrap.cc:654] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.206341 7439 tablet_bootstrap.cc:492] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:53.206406 7439 ts_tablet_manager.cc:1367] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:53.206766 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 85cc7734181b4c89b7d50f9e81a2f583 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=eaceddc0ae4d496fa3ffc653930395b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:53.206795 7439 raft_consensus.cc:348] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:53.206883 7439 raft_consensus.cc:374] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.206908 7439 raft_consensus.cc:729] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.206992 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85cc7734181b4c89b7d50f9e81a2f583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.207013 7439 consensus_queue.cc:260] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:53.207247 7439 ts_tablet_manager.cc:1398] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:53.209417 7442 tablet_bootstrap.cc:492] T 85cc7734181b4c89b7d50f9e81a2f583 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:53.211688 327 tablet_service.cc:1467] Processing CreateTablet for tablet 85cc7734181b4c89b7d50f9e81a2f583 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=eaceddc0ae4d496fa3ffc653930395b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:53.211917 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85cc7734181b4c89b7d50f9e81a2f583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.210736 7442 tablet_bootstrap.cc:654] T 85cc7734181b4c89b7d50f9e81a2f583 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.217080 32533 tablet_service.cc:1467] Processing CreateTablet for tablet bcec6adcdf4e471b86954e6d99528a22 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=eaceddc0ae4d496fa3ffc653930395b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:53.217336 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcec6adcdf4e471b86954e6d99528a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.219627 7445 tablet_bootstrap.cc:492] T bcec6adcdf4e471b86954e6d99528a22 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:53.220458 7445 tablet_bootstrap.cc:654] T bcec6adcdf4e471b86954e6d99528a22 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.223085 7445 tablet_bootstrap.cc:492] T bcec6adcdf4e471b86954e6d99528a22 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:53.223168 7445 ts_tablet_manager.cc:1367] T bcec6adcdf4e471b86954e6d99528a22 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:53.223623 7445 raft_consensus.cc:348] T bcec6adcdf4e471b86954e6d99528a22 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:53.223728 7445 raft_consensus.cc:374] T bcec6adcdf4e471b86954e6d99528a22 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.223755 7445 raft_consensus.cc:729] T bcec6adcdf4e471b86954e6d99528a22 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.223874 7445 consensus_queue.cc:260] T bcec6adcdf4e471b86954e6d99528a22 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:53.224153 7445 ts_tablet_manager.cc:1398] T bcec6adcdf4e471b86954e6d99528a22 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:53.236636 30277 raft_consensus.cc:1203] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:53.237032 7443 tablet_bootstrap.cc:492] T 85cc7734181b4c89b7d50f9e81a2f583 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:53.237120 7437 tablet_bootstrap.cc:492] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:53.238238 7438 tablet_bootstrap.cc:492] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:53.238845 7441 tablet_bootstrap.cc:492] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:53.239991 7441 tablet_bootstrap.cc:654] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.240947 7438 tablet_bootstrap.cc:654] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.241231 7437 tablet_bootstrap.cc:654] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.241829 7443 tablet_bootstrap.cc:654] T 85cc7734181b4c89b7d50f9e81a2f583 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.245244 7442 tablet_bootstrap.cc:492] T 85cc7734181b4c89b7d50f9e81a2f583 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:53.245256 7443 tablet_bootstrap.cc:492] T 85cc7734181b4c89b7d50f9e81a2f583 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:53.245333 7443 ts_tablet_manager.cc:1367] T 85cc7734181b4c89b7d50f9e81a2f583 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:53.245348 7442 ts_tablet_manager.cc:1367] T 85cc7734181b4c89b7d50f9e81a2f583 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20240501 14:09:53.245801 7442 raft_consensus.cc:348] T 85cc7734181b4c89b7d50f9e81a2f583 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:53.245805 7443 raft_consensus.cc:348] T 85cc7734181b4c89b7d50f9e81a2f583 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:53.245903 7443 raft_consensus.cc:374] T 85cc7734181b4c89b7d50f9e81a2f583 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.245903 7442 raft_consensus.cc:374] T 85cc7734181b4c89b7d50f9e81a2f583 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.245944 7443 raft_consensus.cc:729] T 85cc7734181b4c89b7d50f9e81a2f583 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.245962 7442 raft_consensus.cc:729] T 85cc7734181b4c89b7d50f9e81a2f583 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.246130 7442 consensus_queue.cc:260] T 85cc7734181b4c89b7d50f9e81a2f583 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:53.246274 7443 consensus_queue.cc:260] T 85cc7734181b4c89b7d50f9e81a2f583 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:53.246770 7442 ts_tablet_manager.cc:1398] T 85cc7734181b4c89b7d50f9e81a2f583 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:53.251143 7437 tablet_bootstrap.cc:492] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:53.251255 7443 ts_tablet_manager.cc:1398] T 85cc7734181b4c89b7d50f9e81a2f583 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:53.251300 7437 ts_tablet_manager.cc:1367] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:09:53.252089 7437 raft_consensus.cc:348] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:53.256311 7437 raft_consensus.cc:374] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.256403 7437 raft_consensus.cc:729] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.256562 7437 consensus_queue.cc:260] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:53.257275 7437 ts_tablet_manager.cc:1398] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:53.257475 7437 tablet_bootstrap.cc:492] T 42636717a44e4e3782037c48d8913307 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:53.258692 7437 tablet_bootstrap.cc:654] T 42636717a44e4e3782037c48d8913307 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.260164 7441 tablet_bootstrap.cc:492] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:53.260329 7441 ts_tablet_manager.cc:1367] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240501 14:09:53.260778 7441 raft_consensus.cc:348] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:53.260947 7441 raft_consensus.cc:374] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.261014 7441 raft_consensus.cc:729] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.261186 7441 consensus_queue.cc:260] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:53.261768 7441 ts_tablet_manager.cc:1398] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:53.262483 7438 tablet_bootstrap.cc:492] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:53.262624 7438 ts_tablet_manager.cc:1367] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240501 14:09:53.263175 7438 raft_consensus.cc:348] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:53.263262 7437 tablet_bootstrap.cc:492] T 42636717a44e4e3782037c48d8913307 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:53.263303 7438 raft_consensus.cc:374] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.263341 7437 ts_tablet_manager.cc:1367] T 42636717a44e4e3782037c48d8913307 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:53.263377 7438 raft_consensus.cc:729] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.263525 7438 consensus_queue.cc:260] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:53.263798 7438 ts_tablet_manager.cc:1398] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:53.263768 7437 raft_consensus.cc:348] T 42636717a44e4e3782037c48d8913307 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:53.263854 7437 raft_consensus.cc:374] T 42636717a44e4e3782037c48d8913307 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.263934 7437 raft_consensus.cc:729] T 42636717a44e4e3782037c48d8913307 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.264092 7437 consensus_queue.cc:260] T 42636717a44e4e3782037c48d8913307 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:53.264182 7438 tablet_bootstrap.cc:492] T bcec6adcdf4e471b86954e6d99528a22 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:53.264487 7437 ts_tablet_manager.cc:1398] T 42636717a44e4e3782037c48d8913307 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:53.265220 7438 tablet_bootstrap.cc:654] T bcec6adcdf4e471b86954e6d99528a22 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.266338 7438 tablet_bootstrap.cc:492] T bcec6adcdf4e471b86954e6d99528a22 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:53.266508 7438 ts_tablet_manager.cc:1367] T bcec6adcdf4e471b86954e6d99528a22 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:53.266911 7438 raft_consensus.cc:348] T bcec6adcdf4e471b86954e6d99528a22 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:53.267038 7438 raft_consensus.cc:374] T bcec6adcdf4e471b86954e6d99528a22 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.267086 7438 raft_consensus.cc:729] T bcec6adcdf4e471b86954e6d99528a22 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.267195 7438 consensus_queue.cc:260] T bcec6adcdf4e471b86954e6d99528a22 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:53.267570 7438 ts_tablet_manager.cc:1398] T bcec6adcdf4e471b86954e6d99528a22 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:53.289690 30277 consensus_queue.cc:237] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:53.290812 32554 raft_consensus.cc:1261] T 19f203c753ef481dafc091ea2d60f56f P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:53.290956 30014 raft_consensus.cc:1261] T 19f203c753ef481dafc091ea2d60f56f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:53.291314 6686 consensus_queue.cc:1035] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:53.291484 7386 consensus_queue.cc:1035] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:53.293196 6683 raft_consensus.cc:2932] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:53.293658 30014 raft_consensus.cc:2932] T 19f203c753ef481dafc091ea2d60f56f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:53.295346 32554 raft_consensus.cc:2932] T 19f203c753ef481dafc091ea2d60f56f P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:53.295881 29867 catalog_manager.cc:5484] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:53.297724 29871 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:53.299177 30126 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f203c753ef481dafc091ea2d60f56f with delete_type TABLET_DATA_TOMBSTONED (TS d35c16dcb506426993811bb2519d7db3 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:53.299592 7471 tablet_replica.cc:330] T 19f203c753ef481dafc091ea2d60f56f P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:53.299904 7471 raft_consensus.cc:2229] T 19f203c753ef481dafc091ea2d60f56f P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:53.300215 7471 raft_consensus.cc:2258] T 19f203c753ef481dafc091ea2d60f56f P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:53.300992 7471 ts_tablet_manager.cc:1875] T 19f203c753ef481dafc091ea2d60f56f P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:53.303846 7471 ts_tablet_manager.cc:1888] T 19f203c753ef481dafc091ea2d60f56f P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:53.303928 7471 log.cc:1196] T 19f203c753ef481dafc091ea2d60f56f P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/19f203c753ef481dafc091ea2d60f56f
I20240501 14:09:53.304487 7387 tablet.cc:1722] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:53.305305 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 19f203c753ef481dafc091ea2d60f56f (table rebalance_test_table_19 [id=00bf2d6ad5af4e6480096d5ceb4b1bb8]) successfully deleted
I20240501 14:09:53.305426 7394 tablet.cc:1722] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:53.306628 7392 tablet.cc:1722] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:53.310704 29866 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=e43424dadcf24e538efa4828a56d7035] alter complete (version 3)
I20240501 14:09:53.395162 32685 raft_consensus.cc:1203] T f740d894942a4bb584ca5688ef823d55 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:53.400022 6686 raft_consensus.cc:482] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:53.400164 6686 raft_consensus.cc:504] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:53.400637 6686 leader_election.cc:288] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:53.400936 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcec6adcdf4e471b86954e6d99528a22" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:53.400972 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcec6adcdf4e471b86954e6d99528a22" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:53.401069 32554 raft_consensus.cc:2454] T bcec6adcdf4e471b86954e6d99528a22 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:53.401147 30145 raft_consensus.cc:2454] T bcec6adcdf4e471b86954e6d99528a22 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:53.401366 30213 leader_election.cc:302] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 5f8a0cfdfdca4148a75686dc88059cf3; no voters:
I20240501 14:09:53.401670 6686 raft_consensus.cc:2781] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:53.401731 6686 raft_consensus.cc:482] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:53.401777 6686 raft_consensus.cc:3037] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.402817 6686 raft_consensus.cc:504] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:53.403205 6686 leader_election.cc:288] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:53.403720 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcec6adcdf4e471b86954e6d99528a22" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:53.403856 30145 raft_consensus.cc:3037] T bcec6adcdf4e471b86954e6d99528a22 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.404363 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcec6adcdf4e471b86954e6d99528a22" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:53.404459 32554 raft_consensus.cc:3037] T bcec6adcdf4e471b86954e6d99528a22 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.404748 30145 raft_consensus.cc:2454] T bcec6adcdf4e471b86954e6d99528a22 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:53.405025 30213 leader_election.cc:302] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:53.405177 6686 raft_consensus.cc:2781] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:53.405252 6686 raft_consensus.cc:686] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:53.405316 32554 raft_consensus.cc:2454] T bcec6adcdf4e471b86954e6d99528a22 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:53.405370 6686 consensus_queue.cc:237] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:53.407305 29866 catalog_manager.cc:5484] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:53.411767 29866 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:53.411866 29866 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:53.413892 29866 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 3758e4f9cfa145db8438d6be1a7aca0f
I20240501 14:09:53.414431 30126 tablet_service.cc:1514] Processing DeleteTablet for tablet 3758e4f9cfa145db8438d6be1a7aca0f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:53 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:53.414609 7471 tablet_replica.cc:330] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:53.414747 7471 raft_consensus.cc:2229] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:53.414888 7471 raft_consensus.cc:2258] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:53.415483 7471 ts_tablet_manager.cc:1875] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:53.417292 32401 tablet_service.cc:1514] Processing DeleteTablet for tablet 3758e4f9cfa145db8438d6be1a7aca0f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:53 UTC) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:53.417676 7471 ts_tablet_manager.cc:1888] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:53.417742 7471 log.cc:1196] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/3758e4f9cfa145db8438d6be1a7aca0f
I20240501 14:09:53.417800 30257 tablet_service.cc:1514] Processing DeleteTablet for tablet 3758e4f9cfa145db8438d6be1a7aca0f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:53 UTC) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:53.418017 7471 ts_tablet_manager.cc:1909] T 3758e4f9cfa145db8438d6be1a7aca0f P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:53.418169 7515 tablet_replica.cc:330] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:53.418421 7515 raft_consensus.cc:2229] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:53.418653 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 3758e4f9cfa145db8438d6be1a7aca0f (table rebalancer_dynamic_table_new_name [id=e43424dadcf24e538efa4828a56d7035]) successfully deleted
I20240501 14:09:53.418728 7515 raft_consensus.cc:2258] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:53.419262 7515 ts_tablet_manager.cc:1875] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:53.421545 7515 ts_tablet_manager.cc:1888] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:53.421622 7515 log.cc:1196] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/3758e4f9cfa145db8438d6be1a7aca0f
I20240501 14:09:53.421923 7515 ts_tablet_manager.cc:1909] T 3758e4f9cfa145db8438d6be1a7aca0f P 1d0bb69611af4876a2b3b33c4766e13a: Deleting consensus metadata
I20240501 14:09:53.422483 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet 3758e4f9cfa145db8438d6be1a7aca0f (table rebalancer_dynamic_table_new_name [id=e43424dadcf24e538efa4828a56d7035]) successfully deleted
I20240501 14:09:53.424525 7514 tablet_replica.cc:330] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:53.424695 7514 raft_consensus.cc:2229] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:53.424839 7514 raft_consensus.cc:2258] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:53.425382 7514 ts_tablet_manager.cc:1875] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:53.427265 7514 ts_tablet_manager.cc:1888] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:53.427325 7514 log.cc:1196] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/3758e4f9cfa145db8438d6be1a7aca0f
I20240501 14:09:53.427534 7514 ts_tablet_manager.cc:1909] T 3758e4f9cfa145db8438d6be1a7aca0f P a91029b90bf9415a89a2f70736759ad3: Deleting consensus metadata
I20240501 14:09:53.428133 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 3758e4f9cfa145db8438d6be1a7aca0f (table rebalancer_dynamic_table_new_name [id=e43424dadcf24e538efa4828a56d7035]) successfully deleted
I20240501 14:09:53.518425 29866 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:53.523454 327 tablet_service.cc:1467] Processing CreateTablet for tablet 1cda6759887f4f2f80d00d1ca81d586f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a6d02b0fb4fd4bfcbfaefd24a096a802]), partition=
I20240501 14:09:53.523588 30126 tablet_service.cc:1467] Processing CreateTablet for tablet 1cda6759887f4f2f80d00d1ca81d586f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a6d02b0fb4fd4bfcbfaefd24a096a802]), partition=
I20240501 14:09:53.523725 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cda6759887f4f2f80d00d1ca81d586f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.523830 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cda6759887f4f2f80d00d1ca81d586f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.523967 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 1cda6759887f4f2f80d00d1ca81d586f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a6d02b0fb4fd4bfcbfaefd24a096a802]), partition=
I20240501 14:09:53.524271 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cda6759887f4f2f80d00d1ca81d586f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.525966 7443 tablet_bootstrap.cc:492] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:53.526417 7516 tablet_bootstrap.cc:492] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:53.526904 7443 tablet_bootstrap.cc:654] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.527453 7516 tablet_bootstrap.cc:654] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.527540 7438 tablet_bootstrap.cc:492] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:53.528951 7438 tablet_bootstrap.cc:654] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.529032 7516 tablet_bootstrap.cc:492] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:53.529122 7516 ts_tablet_manager.cc:1367] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:53.529615 7516 raft_consensus.cc:348] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:53.529762 7516 raft_consensus.cc:374] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.529809 7516 raft_consensus.cc:729] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.529937 7516 consensus_queue.cc:260] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:53.530325 7443 tablet_bootstrap.cc:492] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:53.530292 7516 ts_tablet_manager.cc:1398] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:53.530411 7443 ts_tablet_manager.cc:1367] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:53.530754 7443 raft_consensus.cc:348] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:53.530850 7443 raft_consensus.cc:374] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.530884 7443 raft_consensus.cc:729] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.530997 7443 consensus_queue.cc:260] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:53.531271 7443 ts_tablet_manager.cc:1398] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:53.532511 7438 tablet_bootstrap.cc:492] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:53.532585 7438 ts_tablet_manager.cc:1367] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:09:53.532974 7438 raft_consensus.cc:348] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:53.533092 7438 raft_consensus.cc:374] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.533123 7438 raft_consensus.cc:729] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.533238 7438 consensus_queue.cc:260] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:53.533512 7438 ts_tablet_manager.cc:1398] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:53.543543 7205 raft_consensus.cc:482] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:53.543653 7205 raft_consensus.cc:504] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:53.544021 7205 leader_election.cc:288] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:53.544332 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cc7734181b4c89b7d50f9e81a2f583" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:53.544370 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cc7734181b4c89b7d50f9e81a2f583" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:53.544452 349 raft_consensus.cc:2454] T 85cc7734181b4c89b7d50f9e81a2f583 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:53.544494 32685 raft_consensus.cc:2454] T 85cc7734181b4c89b7d50f9e81a2f583 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:53.544734 30081 leader_election.cc:302] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:53.544914 7205 raft_consensus.cc:2781] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:53.544966 7205 raft_consensus.cc:482] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:53.545013 7205 raft_consensus.cc:3037] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.545902 7205 raft_consensus.cc:504] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:53.546204 7205 leader_election.cc:288] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:53.546445 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cc7734181b4c89b7d50f9e81a2f583" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:53.546546 349 raft_consensus.cc:3037] T 85cc7734181b4c89b7d50f9e81a2f583 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.546540 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cc7734181b4c89b7d50f9e81a2f583" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:53.546653 32685 raft_consensus.cc:3037] T 85cc7734181b4c89b7d50f9e81a2f583 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.547268 349 raft_consensus.cc:2454] T 85cc7734181b4c89b7d50f9e81a2f583 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:53.547425 32685 raft_consensus.cc:2454] T 85cc7734181b4c89b7d50f9e81a2f583 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:53.547535 30081 leader_election.cc:302] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:53.547684 7205 raft_consensus.cc:2781] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:53.547785 7205 raft_consensus.cc:686] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:53.547923 7205 consensus_queue.cc:237] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:53.550109 29866 catalog_manager.cc:5484] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:53.566505 7382 raft_consensus.cc:482] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:53.566614 7382 raft_consensus.cc:504] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:53.567059 7382 leader_election.cc:288] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:53.567397 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cda6759887f4f2f80d00d1ca81d586f" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:53.567430 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cda6759887f4f2f80d00d1ca81d586f" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:53.567507 30145 raft_consensus.cc:2454] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:53.567574 30013 raft_consensus.cc:2454] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:53.567746 32753 leader_election.cc:302] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:53.567906 7382 raft_consensus.cc:2781] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:53.567984 7382 raft_consensus.cc:482] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:53.568032 7382 raft_consensus.cc:3037] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.568809 7382 raft_consensus.cc:504] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:53.569139 7382 leader_election.cc:288] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:53.569370 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cda6759887f4f2f80d00d1ca81d586f" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:53.569491 30145 raft_consensus.cc:3037] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.569589 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cda6759887f4f2f80d00d1ca81d586f" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:53.569682 30013 raft_consensus.cc:3037] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.570317 30145 raft_consensus.cc:2454] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:53.570564 30013 raft_consensus.cc:2454] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:53.570752 32753 leader_election.cc:302] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:53.570884 7382 raft_consensus.cc:2781] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:53.570964 7382 raft_consensus.cc:686] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:53.571094 7382 consensus_queue.cc:237] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:53.573307 29866 catalog_manager.cc:5484] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:53.633365 6905 raft_consensus.cc:482] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:53.633463 6905 raft_consensus.cc:504] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:53.633868 6185 raft_consensus.cc:482] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:53.633899 6905 leader_election.cc:288] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:53.633961 6185 raft_consensus.cc:504] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:53.634299 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42636717a44e4e3782037c48d8913307" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:53.634297 30408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42636717a44e4e3782037c48d8913307" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:53.634389 6185 leader_election.cc:288] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:53.634418 30264 raft_consensus.cc:2454] T 42636717a44e4e3782037c48d8913307 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:53.634416 30408 raft_consensus.cc:2454] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:53.634692 32358 leader_election.cc:302] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:53.634845 6905 raft_consensus.cc:2781] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:53.634900 6905 raft_consensus.cc:482] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:53.634944 6905 raft_consensus.cc:3037] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.636108 6905 raft_consensus.cc:504] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:53.636385 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42636717a44e4e3782037c48d8913307" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:53.636473 6905 leader_election.cc:288] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:53.636518 32415 raft_consensus.cc:2379] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1c217469ae4478ebd49e4ed9f7352d7 in current term 1: Already voted for candidate a91029b90bf9415a89a2f70736759ad3 in this term.
I20240501 14:09:53.636813 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42636717a44e4e3782037c48d8913307" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:53.636932 30264 raft_consensus.cc:2454] T 42636717a44e4e3782037c48d8913307 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:53.637183 30343 leader_election.cc:302] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters: a91029b90bf9415a89a2f70736759ad3
I20240501 14:09:53.637393 6185 raft_consensus.cc:2781] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:53.637439 6185 raft_consensus.cc:482] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:53.637472 6185 raft_consensus.cc:3037] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.637650 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42636717a44e4e3782037c48d8913307" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:53.637777 30277 raft_consensus.cc:3037] T 42636717a44e4e3782037c48d8913307 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.638432 6185 raft_consensus.cc:504] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:53.638669 30277 raft_consensus.cc:2454] T 42636717a44e4e3782037c48d8913307 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:53.638763 6185 leader_election.cc:288] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:53.638953 32358 leader_election.cc:302] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:53.639405 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42636717a44e4e3782037c48d8913307" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:53.639523 32415 raft_consensus.cc:2379] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d1c217469ae4478ebd49e4ed9f7352d7 in current term 1: Already voted for candidate a91029b90bf9415a89a2f70736759ad3 in this term.
I20240501 14:09:53.640048 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42636717a44e4e3782037c48d8913307" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:53.640197 30277 raft_consensus.cc:2379] T 42636717a44e4e3782037c48d8913307 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d1c217469ae4478ebd49e4ed9f7352d7 in current term 1: Already voted for candidate a91029b90bf9415a89a2f70736759ad3 in this term.
I20240501 14:09:53.641582 30408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42636717a44e4e3782037c48d8913307" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:53.641633 30343 leader_election.cc:302] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7; no voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3
I20240501 14:09:53.641728 30408 raft_consensus.cc:2379] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a91029b90bf9415a89a2f70736759ad3 in current term 1: Already voted for candidate d1c217469ae4478ebd49e4ed9f7352d7 in this term.
I20240501 14:09:53.641785 6185 raft_consensus.cc:2726] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 14:09:53.641883 6905 raft_consensus.cc:2781] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:53.641976 6905 raft_consensus.cc:686] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:53.642113 6905 consensus_queue.cc:237] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:53.644500 29866 catalog_manager.cc:5484] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:53.673064 32401 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:53.673321 30257 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:53.674014 30388 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:53.677388 29866 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:53.677983 29993 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:53.684163 32533 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:53.685014 30126 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:53.701756 327 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:53.706350 32665 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:53.725615 30013 raft_consensus.cc:1261] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:53.726275 6424 consensus_queue.cc:1035] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:53.730188 29866 catalog_manager.cc:5576] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387) does not have the latest schema for tablet 1cda6759887f4f2f80d00d1ca81d586f (table rebalancer_dynamic_table [id=a6d02b0fb4fd4bfcbfaefd24a096a802]). Expected version 1 got 0
I20240501 14:09:53.731730 30145 raft_consensus.cc:1261] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:53.732136 6424 consensus_queue.cc:1035] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:53.734705 7543 tablet.cc:1722] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:53.735093 7543 tablet.cc:1620] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:53.735862 7392 tablet.cc:1722] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:53.736114 7392 tablet.cc:1620] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3: MemRowSet was empty: no flush needed.
I20240501 14:09:53.739045 29866 catalog_manager.cc:5576] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457) does not have the latest schema for tablet 1cda6759887f4f2f80d00d1ca81d586f (table rebalancer_dynamic_table [id=a6d02b0fb4fd4bfcbfaefd24a096a802]). Expected version 1 got 0
I20240501 14:09:53.741446 29864 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=a6d02b0fb4fd4bfcbfaefd24a096a802] alter complete (version 1)
I20240501 14:09:53.741716 7544 tablet.cc:1722] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:53.742039 7544 tablet.cc:1620] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e: MemRowSet was empty: no flush needed.
I20240501 14:09:53.765580 30145 consensus_queue.cc:237] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:53.766791 32685 raft_consensus.cc:1261] T 529f938305214c3ba33f61e06d1acd77 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:53.766794 30408 raft_consensus.cc:1261] T 529f938305214c3ba33f61e06d1acd77 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:53.767128 7561 consensus_queue.cc:1035] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:53.767318 7205 consensus_queue.cc:1035] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:53.768895 6666 raft_consensus.cc:2932] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:53.769491 30408 raft_consensus.cc:2932] T 529f938305214c3ba33f61e06d1acd77 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:53.769737 32685 raft_consensus.cc:2932] T 529f938305214c3ba33f61e06d1acd77 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:53.772521 29864 catalog_manager.cc:5484] T 529f938305214c3ba33f61e06d1acd77 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:53.775244 30257 tablet_service.cc:1514] Processing DeleteTablet for tablet 529f938305214c3ba33f61e06d1acd77 with delete_type TABLET_DATA_TOMBSTONED (TS 1d0bb69611af4876a2b3b33c4766e13a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:53.775475 7515 tablet_replica.cc:330] T 529f938305214c3ba33f61e06d1acd77 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:53.775617 7515 raft_consensus.cc:2229] T 529f938305214c3ba33f61e06d1acd77 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:53.775751 7515 raft_consensus.cc:2258] T 529f938305214c3ba33f61e06d1acd77 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:53.776206 7515 ts_tablet_manager.cc:1875] T 529f938305214c3ba33f61e06d1acd77 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:53.778172 7515 ts_tablet_manager.cc:1888] T 529f938305214c3ba33f61e06d1acd77 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:53.778236 7515 log.cc:1196] T 529f938305214c3ba33f61e06d1acd77 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/529f938305214c3ba33f61e06d1acd77
I20240501 14:09:53.778643 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet 529f938305214c3ba33f61e06d1acd77 (table rebalance_test_table_11 [id=0ceb652d71d942b2a6e0ef7686e7635b]) successfully deleted
I20240501 14:09:53.851446 29864 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:53.857244 7543 tablet.cc:1722] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:53.857592 7543 tablet.cc:1620] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:53.857910 7392 tablet.cc:1722] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:53.858186 7392 tablet.cc:1620] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3: MemRowSet was empty: no flush needed.
I20240501 14:09:53.860160 7544 tablet.cc:1722] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:53.860544 7544 tablet.cc:1620] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e: MemRowSet was empty: no flush needed.
I20240501 14:09:53.864265 29871 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=a6d02b0fb4fd4bfcbfaefd24a096a802] alter complete (version 2)
I20240501 14:09:53.905238 32533 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:53.906502 30388 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:53.907737 29993 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:53.918993 327 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:53.920432 32401 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:53.922046 30126 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:53.936398 32665 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:53.938776 30257 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:53.951260 6686 consensus_queue.cc:1035] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:53.956033 6683 consensus_queue.cc:1035] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:53.967257 29865 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:53.972525 7543 tablet.cc:1722] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:53.973029 7544 tablet.cc:1722] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:53.977128 7392 tablet.cc:1722] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:53.985018 29865 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=a6d02b0fb4fd4bfcbfaefd24a096a802] alter complete (version 3)
I20240501 14:09:54.017221 30277 consensus_queue.cc:237] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:54.018584 32553 raft_consensus.cc:1261] T f2bc06e76e2a41429d5625ebf99e5e5c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:54.018767 30013 raft_consensus.cc:1261] T f2bc06e76e2a41429d5625ebf99e5e5c P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:54.019006 6686 consensus_queue.cc:1035] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:54.019213 6683 consensus_queue.cc:1035] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:54.020864 6683 raft_consensus.cc:2932] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:54.021248 32554 raft_consensus.cc:2932] T f2bc06e76e2a41429d5625ebf99e5e5c P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:54.021909 30013 raft_consensus.cc:2932] T f2bc06e76e2a41429d5625ebf99e5e5c P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:54.024506 29865 catalog_manager.cc:5484] T f2bc06e76e2a41429d5625ebf99e5e5c P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: config changed from index 4 to 5, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:54.027202 30126 tablet_service.cc:1514] Processing DeleteTablet for tablet f2bc06e76e2a41429d5625ebf99e5e5c with delete_type TABLET_DATA_TOMBSTONED (TS d35c16dcb506426993811bb2519d7db3 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:54.027534 7642 tablet_replica.cc:330] T f2bc06e76e2a41429d5625ebf99e5e5c P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:54.027684 7642 raft_consensus.cc:2229] T f2bc06e76e2a41429d5625ebf99e5e5c P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:54.027928 7642 raft_consensus.cc:2258] T f2bc06e76e2a41429d5625ebf99e5e5c P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:54.028815 7642 ts_tablet_manager.cc:1875] T f2bc06e76e2a41429d5625ebf99e5e5c P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:54.031543 7642 ts_tablet_manager.cc:1888] T f2bc06e76e2a41429d5625ebf99e5e5c P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:54.031708 7642 log.cc:1196] T f2bc06e76e2a41429d5625ebf99e5e5c P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/f2bc06e76e2a41429d5625ebf99e5e5c
I20240501 14:09:54.032297 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet f2bc06e76e2a41429d5625ebf99e5e5c (table rebalance_test_table_11 [id=0ceb652d71d942b2a6e0ef7686e7635b]) successfully deleted
I20240501 14:09:54.034493 7205 consensus_queue.cc:1035] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:54.037717 6666 consensus_queue.cc:1035] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:54.088483 29869 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:54.088619 29869 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:54.091056 29869 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 1cda6759887f4f2f80d00d1ca81d586f
I20240501 14:09:54.091682 327 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cda6759887f4f2f80d00d1ca81d586f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:54 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:54.091693 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cda6759887f4f2f80d00d1ca81d586f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:54 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:09:54.091950 30126 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cda6759887f4f2f80d00d1ca81d586f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:54 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:54.092065 7689 tablet_replica.cc:330] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:09:54.092200 7689 raft_consensus.cc:2229] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:54.092337 7642 tablet_replica.cc:330] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:54.092406 7689 raft_consensus.cc:2258] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:54.092481 7642 raft_consensus.cc:2229] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:54.092597 7642 raft_consensus.cc:2258] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:54.092622 7688 tablet_replica.cc:330] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:54.092753 7688 raft_consensus.cc:2229] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:54.092905 7688 raft_consensus.cc:2258] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:54.093367 7689 ts_tablet_manager.cc:1875] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:54.093469 7642 ts_tablet_manager.cc:1875] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:54.095695 7642 ts_tablet_manager.cc:1888] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:54.095757 7642 log.cc:1196] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/1cda6759887f4f2f80d00d1ca81d586f
I20240501 14:09:54.095818 7689 ts_tablet_manager.cc:1888] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:54.095866 7689 log.cc:1196] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/1cda6759887f4f2f80d00d1ca81d586f
I20240501 14:09:54.096040 7642 ts_tablet_manager.cc:1909] T 1cda6759887f4f2f80d00d1ca81d586f P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:54.096695 7688 ts_tablet_manager.cc:1875] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:54.099203 7689 ts_tablet_manager.cc:1909] T 1cda6759887f4f2f80d00d1ca81d586f P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:09:54.100272 7688 ts_tablet_manager.cc:1888] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:54.100752 7688 log.cc:1196] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/1cda6759887f4f2f80d00d1ca81d586f
I20240501 14:09:54.101213 7688 ts_tablet_manager.cc:1909] T 1cda6759887f4f2f80d00d1ca81d586f P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:09:54.101320 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 1cda6759887f4f2f80d00d1ca81d586f (table rebalancer_dynamic_table_new_name [id=a6d02b0fb4fd4bfcbfaefd24a096a802]) successfully deleted
I20240501 14:09:54.101701 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 1cda6759887f4f2f80d00d1ca81d586f (table rebalancer_dynamic_table_new_name [id=a6d02b0fb4fd4bfcbfaefd24a096a802]) successfully deleted
I20240501 14:09:54.102164 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 1cda6759887f4f2f80d00d1ca81d586f (table rebalancer_dynamic_table_new_name [id=a6d02b0fb4fd4bfcbfaefd24a096a802]) successfully deleted
I20240501 14:09:54.107570 6905 consensus_queue.cc:1035] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:54.112507 7305 consensus_queue.cc:1035] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:54.150615 32401 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:54.153460 30257 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:54.156697 327 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:54.157390 30388 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:54.160650 29872 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:54.164011 32533 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:54.173210 29995 tablet_service.cc:1467] Processing CreateTablet for tablet 3dccd7ef69294def8573ea17076bbdfb (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=b0f02750933049cf9e3457c96b5a5534]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.173542 29995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dccd7ef69294def8573ea17076bbdfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.176108 32665 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:54.176641 30257 tablet_service.cc:1467] Processing CreateTablet for tablet e6e4b617c50f4441bded13f9551362a0 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=b0f02750933049cf9e3457c96b5a5534]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.176802 30388 tablet_service.cc:1467] Processing CreateTablet for tablet e6e4b617c50f4441bded13f9551362a0 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=b0f02750933049cf9e3457c96b5a5534]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.176946 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e4b617c50f4441bded13f9551362a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.177068 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e4b617c50f4441bded13f9551362a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.179090 7706 tablet_bootstrap.cc:492] T e6e4b617c50f4441bded13f9551362a0 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:54.173235 29994 tablet_service.cc:1467] Processing CreateTablet for tablet f726c6a4929541c9b8ba7e29b757bb89 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=b0f02750933049cf9e3457c96b5a5534]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.179505 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f726c6a4929541c9b8ba7e29b757bb89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.180580 7706 tablet_bootstrap.cc:654] T e6e4b617c50f4441bded13f9551362a0 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.180927 29993 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:54.181733 7706 tablet_bootstrap.cc:492] T e6e4b617c50f4441bded13f9551362a0 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:54.181826 7706 ts_tablet_manager.cc:1367] T e6e4b617c50f4441bded13f9551362a0 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:54.182209 7706 raft_consensus.cc:348] T e6e4b617c50f4441bded13f9551362a0 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:54.182305 7706 raft_consensus.cc:374] T e6e4b617c50f4441bded13f9551362a0 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.182332 7706 raft_consensus.cc:729] T e6e4b617c50f4441bded13f9551362a0 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.182432 7706 consensus_queue.cc:260] T e6e4b617c50f4441bded13f9551362a0 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:54.182667 7706 ts_tablet_manager.cc:1398] T e6e4b617c50f4441bded13f9551362a0 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:54.185007 30126 tablet_service.cc:1467] Processing CreateTablet for tablet 3dccd7ef69294def8573ea17076bbdfb (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=b0f02750933049cf9e3457c96b5a5534]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.185308 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dccd7ef69294def8573ea17076bbdfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.186035 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 3dccd7ef69294def8573ea17076bbdfb (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=b0f02750933049cf9e3457c96b5a5534]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.186353 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dccd7ef69294def8573ea17076bbdfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.186499 327 tablet_service.cc:1467] Processing CreateTablet for tablet e6e4b617c50f4441bded13f9551362a0 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=b0f02750933049cf9e3457c96b5a5534]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.186764 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e4b617c50f4441bded13f9551362a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.187657 32664 tablet_service.cc:1467] Processing CreateTablet for tablet f726c6a4929541c9b8ba7e29b757bb89 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=b0f02750933049cf9e3457c96b5a5534]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.185017 30125 tablet_service.cc:1467] Processing CreateTablet for tablet f726c6a4929541c9b8ba7e29b757bb89 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=b0f02750933049cf9e3457c96b5a5534]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.188067 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f726c6a4929541c9b8ba7e29b757bb89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.191151 7709 tablet_bootstrap.cc:492] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:54.192209 7709 tablet_bootstrap.cc:654] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.192858 29870 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:54.194402 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f726c6a4929541c9b8ba7e29b757bb89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.196475 7711 tablet_bootstrap.cc:492] T f726c6a4929541c9b8ba7e29b757bb89 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:54.197377 7711 tablet_bootstrap.cc:654] T f726c6a4929541c9b8ba7e29b757bb89 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.198594 7711 tablet_bootstrap.cc:492] T f726c6a4929541c9b8ba7e29b757bb89 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:54.198666 7711 ts_tablet_manager.cc:1367] T f726c6a4929541c9b8ba7e29b757bb89 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:54.199065 7711 raft_consensus.cc:348] T f726c6a4929541c9b8ba7e29b757bb89 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:54.199157 7711 raft_consensus.cc:374] T f726c6a4929541c9b8ba7e29b757bb89 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.199183 7711 raft_consensus.cc:729] T f726c6a4929541c9b8ba7e29b757bb89 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.199296 7711 consensus_queue.cc:260] T f726c6a4929541c9b8ba7e29b757bb89 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:54.199546 7711 ts_tablet_manager.cc:1398] T f726c6a4929541c9b8ba7e29b757bb89 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:54.201119 7711 tablet_bootstrap.cc:492] T 3dccd7ef69294def8573ea17076bbdfb P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:54.201987 7711 tablet_bootstrap.cc:654] T 3dccd7ef69294def8573ea17076bbdfb P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.208110 7711 tablet_bootstrap.cc:492] T 3dccd7ef69294def8573ea17076bbdfb P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:54.208297 7711 ts_tablet_manager.cc:1367] T 3dccd7ef69294def8573ea17076bbdfb P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:54.209400 7711 raft_consensus.cc:348] T 3dccd7ef69294def8573ea17076bbdfb P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:54.209666 7711 raft_consensus.cc:374] T 3dccd7ef69294def8573ea17076bbdfb P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.210009 7711 raft_consensus.cc:729] T 3dccd7ef69294def8573ea17076bbdfb P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.210280 7711 consensus_queue.cc:260] T 3dccd7ef69294def8573ea17076bbdfb P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:54.210608 32401 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae05eda77ea42a386e103097454fe95 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ce53c2bad74142a094606c5f9e904185]), partition=
I20240501 14:09:54.210842 32401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae05eda77ea42a386e103097454fe95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.210920 7711 ts_tablet_manager.cc:1398] T 3dccd7ef69294def8573ea17076bbdfb P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:54.211622 7709 tablet_bootstrap.cc:492] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:54.211721 7709 ts_tablet_manager.cc:1367] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.002s
I20240501 14:09:54.212126 7709 raft_consensus.cc:348] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:54.212220 7709 raft_consensus.cc:374] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.212273 7709 raft_consensus.cc:729] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.212392 7709 consensus_queue.cc:260] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:54.212637 7709 ts_tablet_manager.cc:1398] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:54.212779 7709 tablet_bootstrap.cc:492] T f726c6a4929541c9b8ba7e29b757bb89 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:54.213127 7715 tablet_bootstrap.cc:492] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:54.213606 7709 tablet_bootstrap.cc:654] T f726c6a4929541c9b8ba7e29b757bb89 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.215804 32533 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae05eda77ea42a386e103097454fe95 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ce53c2bad74142a094606c5f9e904185]), partition=
I20240501 14:09:54.216017 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae05eda77ea42a386e103097454fe95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.221496 7715 tablet_bootstrap.cc:654] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.221671 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae05eda77ea42a386e103097454fe95 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ce53c2bad74142a094606c5f9e904185]), partition=
I20240501 14:09:54.221923 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae05eda77ea42a386e103097454fe95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.222479 7709 tablet_bootstrap.cc:492] T f726c6a4929541c9b8ba7e29b757bb89 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:54.222553 7709 ts_tablet_manager.cc:1367] T f726c6a4929541c9b8ba7e29b757bb89 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240501 14:09:54.222930 7709 raft_consensus.cc:348] T f726c6a4929541c9b8ba7e29b757bb89 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:54.223034 7709 raft_consensus.cc:374] T f726c6a4929541c9b8ba7e29b757bb89 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.223063 7709 raft_consensus.cc:729] T f726c6a4929541c9b8ba7e29b757bb89 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.223156 7709 consensus_queue.cc:260] T f726c6a4929541c9b8ba7e29b757bb89 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:54.223403 7709 ts_tablet_manager.cc:1398] T f726c6a4929541c9b8ba7e29b757bb89 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:54.232463 30125 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:54.240790 7710 tablet_bootstrap.cc:492] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:54.241061 7705 tablet_bootstrap.cc:492] T 3dccd7ef69294def8573ea17076bbdfb P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:54.244894 7710 tablet_bootstrap.cc:654] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.245163 7705 tablet_bootstrap.cc:654] T 3dccd7ef69294def8573ea17076bbdfb P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.256572 7710 tablet_bootstrap.cc:492] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:54.256747 7710 ts_tablet_manager.cc:1367] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240501 14:09:54.258224 7710 raft_consensus.cc:348] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:54.258461 7710 raft_consensus.cc:374] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.258538 7710 raft_consensus.cc:729] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.258729 7710 consensus_queue.cc:260] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:54.260076 7710 ts_tablet_manager.cc:1398] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:54.260844 7705 tablet_bootstrap.cc:492] T 3dccd7ef69294def8573ea17076bbdfb P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:54.260987 7705 ts_tablet_manager.cc:1367] T 3dccd7ef69294def8573ea17076bbdfb P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 14:09:54.261456 7705 raft_consensus.cc:348] T 3dccd7ef69294def8573ea17076bbdfb P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:54.261615 7705 raft_consensus.cc:374] T 3dccd7ef69294def8573ea17076bbdfb P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.261680 7705 raft_consensus.cc:729] T 3dccd7ef69294def8573ea17076bbdfb P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.261833 7705 consensus_queue.cc:260] T 3dccd7ef69294def8573ea17076bbdfb P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:54.262285 7705 ts_tablet_manager.cc:1398] T 3dccd7ef69294def8573ea17076bbdfb P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:54.262584 7705 tablet_bootstrap.cc:492] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:54.263763 7705 tablet_bootstrap.cc:654] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.264600 7205 raft_consensus.cc:482] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:54.264705 7205 raft_consensus.cc:504] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:54.265054 7205 leader_election.cc:288] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:54.265483 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dccd7ef69294def8573ea17076bbdfb" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:54.265556 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dccd7ef69294def8573ea17076bbdfb" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:54.265611 32685 raft_consensus.cc:2454] T 3dccd7ef69294def8573ea17076bbdfb P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:54.265717 30013 raft_consensus.cc:2454] T 3dccd7ef69294def8573ea17076bbdfb P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:54.265925 30079 leader_election.cc:302] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d35c16dcb506426993811bb2519d7db3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:54.266181 7205 raft_consensus.cc:2781] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:54.266247 7205 raft_consensus.cc:482] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:54.266283 7205 raft_consensus.cc:3037] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.267320 7205 raft_consensus.cc:504] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:54.267640 7205 leader_election.cc:288] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:54.268147 7717 tablet_bootstrap.cc:492] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:54.268193 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dccd7ef69294def8573ea17076bbdfb" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:54.268333 32685 raft_consensus.cc:3037] T 3dccd7ef69294def8573ea17076bbdfb P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.268453 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dccd7ef69294def8573ea17076bbdfb" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:54.268543 30013 raft_consensus.cc:3037] T 3dccd7ef69294def8573ea17076bbdfb P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.269366 32685 raft_consensus.cc:2454] T 3dccd7ef69294def8573ea17076bbdfb P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:54.269549 30013 raft_consensus.cc:2454] T 3dccd7ef69294def8573ea17076bbdfb P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:54.270330 30079 leader_election.cc:302] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d35c16dcb506426993811bb2519d7db3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:54.270821 7707 tablet_bootstrap.cc:492] T e6e4b617c50f4441bded13f9551362a0 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:54.271823 7707 tablet_bootstrap.cc:654] T e6e4b617c50f4441bded13f9551362a0 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.273028 7717 tablet_bootstrap.cc:654] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.274191 7717 tablet_bootstrap.cc:492] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:54.274338 7717 ts_tablet_manager.cc:1367] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:54.274878 6666 raft_consensus.cc:2781] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:54.274847 7717 raft_consensus.cc:348] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:54.274960 6666 raft_consensus.cc:686] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:54.274971 7717 raft_consensus.cc:374] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.275004 7717 raft_consensus.cc:729] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.275108 7717 consensus_queue.cc:260] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:54.275099 6666 consensus_queue.cc:237] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:54.275461 7717 ts_tablet_manager.cc:1398] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:54.276898 7705 tablet_bootstrap.cc:492] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:54.276993 7705 ts_tablet_manager.cc:1367] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:09:54.277446 7705 raft_consensus.cc:348] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:54.277571 7705 raft_consensus.cc:374] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.277607 7705 raft_consensus.cc:729] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.277902 7705 consensus_queue.cc:260] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:54.278255 7705 ts_tablet_manager.cc:1398] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:54.278594 29865 catalog_manager.cc:5484] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:54.282385 7715 tablet_bootstrap.cc:492] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:54.282485 7715 ts_tablet_manager.cc:1367] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.069s user 0.002s sys 0.000s
I20240501 14:09:54.283016 7715 raft_consensus.cc:348] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:54.283126 7715 raft_consensus.cc:374] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.283159 7715 raft_consensus.cc:729] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.283272 7715 consensus_queue.cc:260] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:54.283558 7715 ts_tablet_manager.cc:1398] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:54.287262 7707 tablet_bootstrap.cc:492] T e6e4b617c50f4441bded13f9551362a0 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:54.287358 7707 ts_tablet_manager.cc:1367] T e6e4b617c50f4441bded13f9551362a0 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 14:09:54.287757 7707 raft_consensus.cc:348] T e6e4b617c50f4441bded13f9551362a0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:54.287868 7707 raft_consensus.cc:374] T e6e4b617c50f4441bded13f9551362a0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.287902 7707 raft_consensus.cc:729] T e6e4b617c50f4441bded13f9551362a0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.288030 7707 consensus_queue.cc:260] T e6e4b617c50f4441bded13f9551362a0 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:54.288352 7707 ts_tablet_manager.cc:1398] T e6e4b617c50f4441bded13f9551362a0 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:54.288534 7707 tablet_bootstrap.cc:492] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:54.289481 7707 tablet_bootstrap.cc:654] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.293346 7707 tablet_bootstrap.cc:492] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:54.293429 7707 ts_tablet_manager.cc:1367] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:54.294020 7707 raft_consensus.cc:348] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:54.294190 7707 raft_consensus.cc:374] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.294277 7707 raft_consensus.cc:729] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.294453 7707 consensus_queue.cc:260] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:54.295769 7707 ts_tablet_manager.cc:1398] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:54.303032 7641 raft_consensus.cc:482] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:54.303143 7641 raft_consensus.cc:504] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:54.303586 7641 leader_election.cc:288] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:54.303948 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f726c6a4929541c9b8ba7e29b757bb89" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:54.303992 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f726c6a4929541c9b8ba7e29b757bb89" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:54.304095 30145 raft_consensus.cc:2454] T f726c6a4929541c9b8ba7e29b757bb89 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:54.304121 32685 raft_consensus.cc:2454] T f726c6a4929541c9b8ba7e29b757bb89 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:54.304387 29951 leader_election.cc:302] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:54.304594 7641 raft_consensus.cc:2781] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:54.304642 7641 raft_consensus.cc:482] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:54.304687 7641 raft_consensus.cc:3037] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.305748 7641 raft_consensus.cc:504] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:54.306143 7641 leader_election.cc:288] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:54.306566 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f726c6a4929541c9b8ba7e29b757bb89" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:54.306566 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f726c6a4929541c9b8ba7e29b757bb89" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:54.306679 32685 raft_consensus.cc:3037] T f726c6a4929541c9b8ba7e29b757bb89 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.306679 30145 raft_consensus.cc:3037] T f726c6a4929541c9b8ba7e29b757bb89 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.307627 32685 raft_consensus.cc:2454] T f726c6a4929541c9b8ba7e29b757bb89 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:54.307627 30145 raft_consensus.cc:2454] T f726c6a4929541c9b8ba7e29b757bb89 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:54.307912 29948 leader_election.cc:302] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:54.308069 7641 raft_consensus.cc:2781] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:54.308163 7641 raft_consensus.cc:686] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:54.308334 7641 consensus_queue.cc:237] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:54.310326 29870 catalog_manager.cc:5484] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:54.313159 348 consensus_queue.cc:237] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:54.314242 30145 raft_consensus.cc:1261] T b66c71cbccfa4c96bdf2b438938bf56b P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:54.314317 30013 raft_consensus.cc:1261] T b66c71cbccfa4c96bdf2b438938bf56b P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:54.314589 6424 consensus_queue.cc:1035] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:54.314780 7622 consensus_queue.cc:1035] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:54.316404 7382 raft_consensus.cc:2932] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:54.317363 30013 raft_consensus.cc:2932] T b66c71cbccfa4c96bdf2b438938bf56b P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:54.318723 30145 raft_consensus.cc:2932] T b66c71cbccfa4c96bdf2b438938bf56b P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:54.321087 29872 catalog_manager.cc:5484] T b66c71cbccfa4c96bdf2b438938bf56b P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:54.324662 32533 tablet_service.cc:1514] Processing DeleteTablet for tablet b66c71cbccfa4c96bdf2b438938bf56b with delete_type TABLET_DATA_TOMBSTONED (TS 5f8a0cfdfdca4148a75686dc88059cf3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:54.325021 7745 tablet_replica.cc:330] T b66c71cbccfa4c96bdf2b438938bf56b P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:54.325189 7745 raft_consensus.cc:2229] T b66c71cbccfa4c96bdf2b438938bf56b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:54.325572 7745 raft_consensus.cc:2258] T b66c71cbccfa4c96bdf2b438938bf56b P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:54.326090 7745 ts_tablet_manager.cc:1875] T b66c71cbccfa4c96bdf2b438938bf56b P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:54.328903 7745 ts_tablet_manager.cc:1888] T b66c71cbccfa4c96bdf2b438938bf56b P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:09:54.329034 7745 log.cc:1196] T b66c71cbccfa4c96bdf2b438938bf56b P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/b66c71cbccfa4c96bdf2b438938bf56b
I20240501 14:09:54.329634 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet b66c71cbccfa4c96bdf2b438938bf56b (table rebalancer_extra_table_12 [id=dcad19a5572a46a992d9c7fd61cfcf2b]) successfully deleted
I20240501 14:09:54.355522 30145 consensus_queue.cc:237] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:54.357046 30277 raft_consensus.cc:1261] T 3125c67fe40641c2bfd5dcac81e7237f P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:54.357326 30406 raft_consensus.cc:1261] T 3125c67fe40641c2bfd5dcac81e7237f P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:54.357350 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3125c67fe40641c2bfd5dcac81e7237f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:54.357661 6666 consensus_queue.cc:1035] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:54.357820 7561 consensus_queue.cc:1035] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:54.360286 6666 raft_consensus.cc:2932] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:54.360670 30277 raft_consensus.cc:2932] T 3125c67fe40641c2bfd5dcac81e7237f P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:54.362356 30406 raft_consensus.cc:2932] T 3125c67fe40641c2bfd5dcac81e7237f P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:54.363536 29865 catalog_manager.cc:5484] T 3125c67fe40641c2bfd5dcac81e7237f P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: config changed from index -1 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:54.373085 7655 raft_consensus.cc:482] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:54.373170 7655 raft_consensus.cc:504] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:54.373574 7655 leader_election.cc:288] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:54.373827 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae05eda77ea42a386e103097454fe95" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:54.373916 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae05eda77ea42a386e103097454fe95" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:54.373960 32415 raft_consensus.cc:2454] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:54.374056 32554 raft_consensus.cc:2454] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:54.374183 30343 leader_election.cc:302] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:54.374372 7655 raft_consensus.cc:2781] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:54.374423 7655 raft_consensus.cc:482] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:54.374475 7655 raft_consensus.cc:3037] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.375399 7655 raft_consensus.cc:504] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:54.375679 7655 leader_election.cc:288] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:54.375977 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae05eda77ea42a386e103097454fe95" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:54.376091 32415 raft_consensus.cc:3037] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.376629 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae05eda77ea42a386e103097454fe95" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:54.376731 32554 raft_consensus.cc:3037] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.377465 32415 raft_consensus.cc:2454] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:54.377794 32554 raft_consensus.cc:2454] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:54.378075 30344 leader_election.cc:302] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:54.378340 7655 raft_consensus.cc:2781] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:54.378432 7655 raft_consensus.cc:686] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:54.378544 7655 consensus_queue.cc:237] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:54.380654 29870 catalog_manager.cc:5484] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:54.385903 30406 consensus_queue.cc:237] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:54.387199 30145 raft_consensus.cc:1261] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:54.387359 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cfc701c19cd74d0db2f90ce580230177. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:54.387538 6185 consensus_queue.cc:1035] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:54.388032 30277 raft_consensus.cc:1261] T cfc701c19cd74d0db2f90ce580230177 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:54.388417 6185 consensus_queue.cc:1035] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:54.390385 7621 raft_consensus.cc:2932] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:54.390764 30277 raft_consensus.cc:2932] T cfc701c19cd74d0db2f90ce580230177 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:54.392493 30145 raft_consensus.cc:2932] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:54.393811 29864 catalog_manager.cc:5484] T cfc701c19cd74d0db2f90ce580230177 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: config changed from index -1 to 4, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:54.441097 6424 raft_consensus.cc:482] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:54.441219 6424 raft_consensus.cc:504] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:54.441679 6424 leader_election.cc:288] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:54.442293 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e4b617c50f4441bded13f9551362a0" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:54.442448 30277 raft_consensus.cc:2454] T e6e4b617c50f4441bded13f9551362a0 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:54.442734 32752 leader_election.cc:302] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:54.443357 6424 raft_consensus.cc:2781] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:54.443418 6424 raft_consensus.cc:482] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:54.443379 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e4b617c50f4441bded13f9551362a0" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:54.443468 6424 raft_consensus.cc:3037] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.443504 30406 raft_consensus.cc:2454] T e6e4b617c50f4441bded13f9551362a0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:54.444414 6424 raft_consensus.cc:504] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:54.444788 6424 leader_election.cc:288] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:54.445072 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e4b617c50f4441bded13f9551362a0" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:54.445183 30277 raft_consensus.cc:3037] T e6e4b617c50f4441bded13f9551362a0 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.445257 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e4b617c50f4441bded13f9551362a0" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:54.445367 30406 raft_consensus.cc:3037] T e6e4b617c50f4441bded13f9551362a0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.446236 30277 raft_consensus.cc:2454] T e6e4b617c50f4441bded13f9551362a0 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:54.446342 30406 raft_consensus.cc:2454] T e6e4b617c50f4441bded13f9551362a0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:54.448189 32752 leader_election.cc:302] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:54.449573 6424 raft_consensus.cc:2781] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:54.449666 6424 raft_consensus.cc:686] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:54.449788 6424 consensus_queue.cc:237] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:54.455412 29867 catalog_manager.cc:5484] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:54.483732 29867 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:54.490352 32554 raft_consensus.cc:1261] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:54.490933 7655 consensus_queue.cc:1035] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:54.494438 32415 raft_consensus.cc:1261] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:54.496405 29871 catalog_manager.cc:5576] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497) does not have the latest schema for tablet 5ae05eda77ea42a386e103097454fe95 (table rebalancer_dynamic_table [id=ce53c2bad74142a094606c5f9e904185]). Expected version 1 got 0
I20240501 14:09:54.496652 6185 consensus_queue.cc:1035] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:54.499085 29864 catalog_manager.cc:5576] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497) does not have the latest schema for tablet 5ae05eda77ea42a386e103097454fe95 (table rebalancer_dynamic_table [id=ce53c2bad74142a094606c5f9e904185]). Expected version 1 got 0
I20240501 14:09:54.499877 7838 tablet.cc:1722] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:54.500298 29869 catalog_manager.cc:5576] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265) does not have the latest schema for tablet 5ae05eda77ea42a386e103097454fe95 (table rebalancer_dynamic_table [id=ce53c2bad74142a094606c5f9e904185]). Expected version 1 got 0
I20240501 14:09:54.500617 7840 tablet.cc:1722] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:54.503888 7840 tablet.cc:1620] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:54.501106 7838 tablet.cc:1620] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7: MemRowSet was empty: no flush needed.
I20240501 14:09:54.504930 7841 tablet.cc:1722] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:54.505328 7841 tablet.cc:1620] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:54.512614 29870 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=ce53c2bad74142a094606c5f9e904185] alter complete (version 1)
I20240501 14:09:54.522441 30257 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:54.528424 32533 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:54.539633 32665 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:54.540374 30388 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:54.540611 29994 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:54.544237 32401 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:54.547048 30125 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:54.563808 327 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:54.616425 29873 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:54.622583 7838 tablet.cc:1722] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:54.622856 7841 tablet.cc:1722] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:54.622915 7838 tablet.cc:1620] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7: MemRowSet was empty: no flush needed.
I20240501 14:09:54.623132 7841 tablet.cc:1620] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:54.624789 7840 tablet.cc:1722] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:54.625082 7840 tablet.cc:1620] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:54.630156 29873 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=ce53c2bad74142a094606c5f9e904185] alter complete (version 2)
I20240501 14:09:54.728724 6666 consensus_queue.cc:1035] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:54.734341 29870 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:54.735481 7561 consensus_queue.cc:1035] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:54.745149 32533 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:54.749374 7838 tablet.cc:1722] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:54.749840 7840 tablet.cc:1722] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:54.755566 30257 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:54.755825 29864 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=ce53c2bad74142a094606c5f9e904185] alter complete (version 3)
I20240501 14:09:54.760917 7841 tablet.cc:1722] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:54.765093 327 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:54.768394 30388 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:54.769347 32401 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:54.781229 30125 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:54.783712 7911 ts_tablet_manager.cc:902] T cfc701c19cd74d0db2f90ce580230177 P 5f8a0cfdfdca4148a75686dc88059cf3: Initiating tablet copy from peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:54.784538 7911 tablet_copy_client.cc:308] T cfc701c19cd74d0db2f90ce580230177 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.132:34357
I20240501 14:09:54.785079 30428 tablet_copy_service.cc:140] P d1c217469ae4478ebd49e4ed9f7352d7: Received BeginTabletCopySession request for tablet cfc701c19cd74d0db2f90ce580230177 from peer 5f8a0cfdfdca4148a75686dc88059cf3 ({username='slave'} at 127.22.252.134:53955)
I20240501 14:09:54.785187 30428 tablet_copy_service.cc:161] P d1c217469ae4478ebd49e4ed9f7352d7: Beginning new tablet copy session on tablet cfc701c19cd74d0db2f90ce580230177 from peer 5f8a0cfdfdca4148a75686dc88059cf3 at {username='slave'} at 127.22.252.134:53955: session id = 5f8a0cfdfdca4148a75686dc88059cf3-cfc701c19cd74d0db2f90ce580230177
I20240501 14:09:54.786165 30428 tablet_copy_source_session.cc:205] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:54.787345 7911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc701c19cd74d0db2f90ce580230177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.794037 7911 tablet_copy_client.cc:783] T cfc701c19cd74d0db2f90ce580230177 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:54.798942 29994 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:54.807991 7911 tablet_copy_client.cc:647] T cfc701c19cd74d0db2f90ce580230177 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:54.811684 7911 tablet_copy_client.cc:523] T cfc701c19cd74d0db2f90ce580230177 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:54.814277 7911 tablet_bootstrap.cc:492] T cfc701c19cd74d0db2f90ce580230177 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:54.817363 32665 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:54.819221 7641 consensus_queue.cc:1035] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:54.828024 7378 consensus_queue.cc:1035] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:54.857637 29870 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:54.857801 29870 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:54.859954 29870 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 5ae05eda77ea42a386e103097454fe95
I20240501 14:09:54.860488 32533 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae05eda77ea42a386e103097454fe95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:54 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:54.860563 32401 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae05eda77ea42a386e103097454fe95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:54 UTC) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:54.860785 7919 tablet_replica.cc:330] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:54.860918 7919 raft_consensus.cc:2229] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:54.861027 7919 raft_consensus.cc:2258] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:54.861387 30388 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae05eda77ea42a386e103097454fe95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:54 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:54.861795 7920 tablet_replica.cc:330] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:54.861928 7920 raft_consensus.cc:2229] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:54.862033 7920 raft_consensus.cc:2258] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:54.862615 7920 ts_tablet_manager.cc:1875] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:54.865979 7911 tablet_bootstrap.cc:492] T cfc701c19cd74d0db2f90ce580230177 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:54.866016 7920 ts_tablet_manager.cc:1888] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:54.866071 7920 log.cc:1196] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/5ae05eda77ea42a386e103097454fe95
I20240501 14:09:54.866272 7911 tablet_bootstrap.cc:492] T cfc701c19cd74d0db2f90ce580230177 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap complete.
I20240501 14:09:54.866374 7920 ts_tablet_manager.cc:1909] T 5ae05eda77ea42a386e103097454fe95 P a91029b90bf9415a89a2f70736759ad3: Deleting consensus metadata
I20240501 14:09:54.866442 7911 ts_tablet_manager.cc:1367] T cfc701c19cd74d0db2f90ce580230177 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.052s user 0.021s sys 0.000s
I20240501 14:09:54.866832 7911 raft_consensus.cc:348] T cfc701c19cd74d0db2f90ce580230177 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:54.866961 7911 raft_consensus.cc:729] T cfc701c19cd74d0db2f90ce580230177 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.867112 7911 consensus_queue.cc:260] T cfc701c19cd74d0db2f90ce580230177 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:54.867131 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 5ae05eda77ea42a386e103097454fe95 (table rebalancer_dynamic_table_new_name [id=ce53c2bad74142a094606c5f9e904185]) successfully deleted
I20240501 14:09:54.867406 7911 ts_tablet_manager.cc:1398] T cfc701c19cd74d0db2f90ce580230177 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 14:09:54.867961 30428 tablet_copy_service.cc:342] P d1c217469ae4478ebd49e4ed9f7352d7: Request end of tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-cfc701c19cd74d0db2f90ce580230177 received from {username='slave'} at 127.22.252.134:53955
I20240501 14:09:54.868042 30428 tablet_copy_service.cc:433] P d1c217469ae4478ebd49e4ed9f7352d7: ending tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-cfc701c19cd74d0db2f90ce580230177 on tablet cfc701c19cd74d0db2f90ce580230177 with peer 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:09:54.868621 7919 ts_tablet_manager.cc:1875] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:54.870604 7919 ts_tablet_manager.cc:1888] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:54.870677 7919 log.cc:1196] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/5ae05eda77ea42a386e103097454fe95
I20240501 14:09:54.870949 7919 ts_tablet_manager.cc:1909] T 5ae05eda77ea42a386e103097454fe95 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:54.871541 7921 tablet_replica.cc:330] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:54.871542 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet 5ae05eda77ea42a386e103097454fe95 (table rebalancer_dynamic_table_new_name [id=ce53c2bad74142a094606c5f9e904185]) successfully deleted
I20240501 14:09:54.871696 7921 raft_consensus.cc:2229] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:54.871902 7921 raft_consensus.cc:2258] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:54.872418 7921 ts_tablet_manager.cc:1875] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:54.875306 7921 ts_tablet_manager.cc:1888] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:54.875389 7921 log.cc:1196] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/5ae05eda77ea42a386e103097454fe95
I20240501 14:09:54.875698 7921 ts_tablet_manager.cc:1909] T 5ae05eda77ea42a386e103097454fe95 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:54.876593 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 5ae05eda77ea42a386e103097454fe95 (table rebalancer_dynamic_table_new_name [id=ce53c2bad74142a094606c5f9e904185]) successfully deleted
I20240501 14:09:54.962898 29873 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:54.962909 29870 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:54.971053 7963 ts_tablet_manager.cc:902] T 3125c67fe40641c2bfd5dcac81e7237f P db935501e9104de5b15c8780681d6c38: Initiating tablet copy from peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:54.971714 7963 tablet_copy_client.cc:308] T 3125c67fe40641c2bfd5dcac81e7237f P db935501e9104de5b15c8780681d6c38: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.130:36457
I20240501 14:09:54.972678 30257 tablet_service.cc:1467] Processing CreateTablet for tablet 6b944c93f521418baf4c3a65635fbeca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f14fb343f1104929b44c1fbcfb443ad5]), partition=
I20240501 14:09:54.972935 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b944c93f521418baf4c3a65635fbeca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.972925 32401 tablet_service.cc:1467] Processing CreateTablet for tablet 6b944c93f521418baf4c3a65635fbeca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f14fb343f1104929b44c1fbcfb443ad5]), partition=
I20240501 14:09:54.973150 32401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b944c93f521418baf4c3a65635fbeca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.975593 32533 tablet_service.cc:1467] Processing CreateTablet for tablet 6b944c93f521418baf4c3a65635fbeca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f14fb343f1104929b44c1fbcfb443ad5]), partition=
I20240501 14:09:54.975786 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b944c93f521418baf4c3a65635fbeca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.976411 7970 tablet_bootstrap.cc:492] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:54.976732 32401 tablet_service.cc:1467] Processing CreateTablet for tablet 59b8b59d67c64e2d828aaaa93647a5a4 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=8f130519999a4055a609c067e3e6579f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.976934 32401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59b8b59d67c64e2d828aaaa93647a5a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.976958 30165 tablet_copy_service.cc:140] P d35c16dcb506426993811bb2519d7db3: Received BeginTabletCopySession request for tablet 3125c67fe40641c2bfd5dcac81e7237f from peer db935501e9104de5b15c8780681d6c38 ({username='slave'} at 127.22.252.135:49787)
I20240501 14:09:54.977061 30165 tablet_copy_service.cc:161] P d35c16dcb506426993811bb2519d7db3: Beginning new tablet copy session on tablet 3125c67fe40641c2bfd5dcac81e7237f from peer db935501e9104de5b15c8780681d6c38 at {username='slave'} at 127.22.252.135:49787: session id = db935501e9104de5b15c8780681d6c38-3125c67fe40641c2bfd5dcac81e7237f
I20240501 14:09:54.977898 30165 tablet_copy_source_session.cc:205] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:54.978575 30255 tablet_service.cc:1467] Processing CreateTablet for tablet d6776209de5e405b982f863faa556727 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=8f130519999a4055a609c067e3e6579f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.978612 327 tablet_service.cc:1467] Processing CreateTablet for tablet a4462d7eaaa54223998369121761acd6 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=8f130519999a4055a609c067e3e6579f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.978816 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6776209de5e405b982f863faa556727. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.978874 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4462d7eaaa54223998369121761acd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.979187 32400 tablet_service.cc:1467] Processing CreateTablet for tablet d6776209de5e405b982f863faa556727 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=8f130519999a4055a609c067e3e6579f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.979379 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6776209de5e405b982f863faa556727. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.980468 32534 tablet_service.cc:1467] Processing CreateTablet for tablet d6776209de5e405b982f863faa556727 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=8f130519999a4055a609c067e3e6579f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.980491 32532 tablet_service.cc:1467] Processing CreateTablet for tablet a4462d7eaaa54223998369121761acd6 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=8f130519999a4055a609c067e3e6579f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.980697 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6776209de5e405b982f863faa556727. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.980808 32532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4462d7eaaa54223998369121761acd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.982285 7963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3125c67fe40641c2bfd5dcac81e7237f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.983090 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 59b8b59d67c64e2d828aaaa93647a5a4 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=8f130519999a4055a609c067e3e6579f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.983367 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59b8b59d67c64e2d828aaaa93647a5a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.984691 7963 tablet_copy_client.cc:783] T 3125c67fe40641c2bfd5dcac81e7237f P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:54.984829 7963 tablet_copy_client.cc:647] T 3125c67fe40641c2bfd5dcac81e7237f P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:54.986109 7972 tablet_bootstrap.cc:492] T a4462d7eaaa54223998369121761acd6 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:54.987342 7971 tablet_bootstrap.cc:492] T a4462d7eaaa54223998369121761acd6 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:54.987423 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 59b8b59d67c64e2d828aaaa93647a5a4 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=8f130519999a4055a609c067e3e6579f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.987670 7970 tablet_bootstrap.cc:654] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.987687 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59b8b59d67c64e2d828aaaa93647a5a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.988343 7971 tablet_bootstrap.cc:654] T a4462d7eaaa54223998369121761acd6 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.988767 7970 tablet_bootstrap.cc:492] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:54.988855 7970 ts_tablet_manager.cc:1367] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:09:54.988902 7972 tablet_bootstrap.cc:654] T a4462d7eaaa54223998369121761acd6 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.989367 7970 raft_consensus.cc:348] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:54.989465 7970 raft_consensus.cc:374] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.989495 7970 raft_consensus.cc:729] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.989615 7970 consensus_queue.cc:260] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:54.989907 7970 ts_tablet_manager.cc:1398] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:54.990064 7970 tablet_bootstrap.cc:492] T 59b8b59d67c64e2d828aaaa93647a5a4 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:54.990932 7972 tablet_bootstrap.cc:492] T a4462d7eaaa54223998369121761acd6 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:54.990962 7970 tablet_bootstrap.cc:654] T 59b8b59d67c64e2d828aaaa93647a5a4 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.991009 7972 ts_tablet_manager.cc:1367] T a4462d7eaaa54223998369121761acd6 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:54.992590 7972 raft_consensus.cc:348] T a4462d7eaaa54223998369121761acd6 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:54.992688 7963 tablet_copy_client.cc:523] T 3125c67fe40641c2bfd5dcac81e7237f P db935501e9104de5b15c8780681d6c38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:54.992738 7972 raft_consensus.cc:374] T a4462d7eaaa54223998369121761acd6 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.992779 7972 raft_consensus.cc:729] T a4462d7eaaa54223998369121761acd6 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.992898 7972 consensus_queue.cc:260] T a4462d7eaaa54223998369121761acd6 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:54.993168 7972 ts_tablet_manager.cc:1398] T a4462d7eaaa54223998369121761acd6 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:54.993993 7963 tablet_bootstrap.cc:492] T 3125c67fe40641c2bfd5dcac81e7237f P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:54.994699 7971 tablet_bootstrap.cc:492] T a4462d7eaaa54223998369121761acd6 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:54.994777 7971 ts_tablet_manager.cc:1367] T a4462d7eaaa54223998369121761acd6 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240501 14:09:54.995285 7971 raft_consensus.cc:348] T a4462d7eaaa54223998369121761acd6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:54.995376 7971 raft_consensus.cc:374] T a4462d7eaaa54223998369121761acd6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.995412 7971 raft_consensus.cc:729] T a4462d7eaaa54223998369121761acd6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.995532 7971 consensus_queue.cc:260] T a4462d7eaaa54223998369121761acd6 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:54.995818 7971 ts_tablet_manager.cc:1398] T a4462d7eaaa54223998369121761acd6 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:54.996117 7971 tablet_bootstrap.cc:492] T d6776209de5e405b982f863faa556727 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:54.997009 7971 tablet_bootstrap.cc:654] T d6776209de5e405b982f863faa556727 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.998106 7971 tablet_bootstrap.cc:492] T d6776209de5e405b982f863faa556727 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:54.998307 7971 ts_tablet_manager.cc:1367] T d6776209de5e405b982f863faa556727 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:54.998802 7971 raft_consensus.cc:348] T d6776209de5e405b982f863faa556727 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:54.998950 7971 raft_consensus.cc:374] T d6776209de5e405b982f863faa556727 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.999064 7971 raft_consensus.cc:729] T d6776209de5e405b982f863faa556727 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.999289 7971 consensus_queue.cc:260] T d6776209de5e405b982f863faa556727 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:54.999696 7853 raft_consensus.cc:482] T d6776209de5e405b982f863faa556727 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:54.999778 7853 raft_consensus.cc:504] T d6776209de5e405b982f863faa556727 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:55.000797 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6776209de5e405b982f863faa556727" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:55.001143 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6776209de5e405b982f863faa556727" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:55.000202 7853 leader_election.cc:288] T d6776209de5e405b982f863faa556727 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:55.000936 7971 ts_tablet_manager.cc:1398] T d6776209de5e405b982f863faa556727 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20240501 14:09:55.001184 32489 leader_election.cc:341] T d6776209de5e405b982f863faa556727 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:09:55.002349 32489 leader_election.cc:341] T d6776209de5e405b982f863faa556727 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:55.002422 32489 leader_election.cc:302] T d6776209de5e405b982f863faa556727 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3; no voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3
I20240501 14:09:55.002707 7853 raft_consensus.cc:2726] T d6776209de5e405b982f863faa556727 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:09:55.003388 7971 tablet_bootstrap.cc:492] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:55.004175 7970 tablet_bootstrap.cc:492] T 59b8b59d67c64e2d828aaaa93647a5a4 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:55.004302 7970 ts_tablet_manager.cc:1367] T 59b8b59d67c64e2d828aaaa93647a5a4 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:09:55.004755 7970 raft_consensus.cc:348] T 59b8b59d67c64e2d828aaaa93647a5a4 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:55.004877 7970 raft_consensus.cc:374] T 59b8b59d67c64e2d828aaaa93647a5a4 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.004914 7970 raft_consensus.cc:729] T 59b8b59d67c64e2d828aaaa93647a5a4 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.005033 7970 consensus_queue.cc:260] T 59b8b59d67c64e2d828aaaa93647a5a4 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:55.005321 7970 ts_tablet_manager.cc:1398] T 59b8b59d67c64e2d828aaaa93647a5a4 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:55.005496 7970 tablet_bootstrap.cc:492] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:55.005968 7971 tablet_bootstrap.cc:654] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.006435 7970 tablet_bootstrap.cc:654] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.009717 30257 tablet_service.cc:1467] Processing CreateTablet for tablet a4462d7eaaa54223998369121761acd6 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=8f130519999a4055a609c067e3e6579f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:55.009932 7970 tablet_bootstrap.cc:492] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:55.009984 7976 tablet_bootstrap.cc:492] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:55.009987 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4462d7eaaa54223998369121761acd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.010052 7970 ts_tablet_manager.cc:1367] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:55.010521 7970 raft_consensus.cc:348] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:55.010627 7970 raft_consensus.cc:374] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.010658 7970 raft_consensus.cc:729] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.010767 7970 consensus_queue.cc:260] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:55.010902 7976 tablet_bootstrap.cc:654] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.011052 7970 ts_tablet_manager.cc:1398] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:55.011729 7973 tablet_bootstrap.cc:492] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:55.011844 7971 tablet_bootstrap.cc:492] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:55.011955 7971 ts_tablet_manager.cc:1367] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240501 14:09:55.012415 7971 raft_consensus.cc:348] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:55.012513 7971 raft_consensus.cc:374] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.012545 7971 raft_consensus.cc:729] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.012647 7971 consensus_queue.cc:260] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:55.012902 7971 ts_tablet_manager.cc:1398] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:55.013187 7973 tablet_bootstrap.cc:654] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.016510 7976 tablet_bootstrap.cc:492] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:55.016616 7976 ts_tablet_manager.cc:1367] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:55.017063 7976 raft_consensus.cc:348] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:55.017172 7976 raft_consensus.cc:374] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.017203 7976 raft_consensus.cc:729] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.017325 7976 consensus_queue.cc:260] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:55.017632 7976 ts_tablet_manager.cc:1398] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:55.019335 7969 tablet_bootstrap.cc:492] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:55.021728 7969 tablet_bootstrap.cc:654] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.023726 7973 tablet_bootstrap.cc:492] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:55.023873 7973 ts_tablet_manager.cc:1367] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:55.024554 7973 raft_consensus.cc:348] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:55.025024 7973 raft_consensus.cc:374] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.025120 7973 raft_consensus.cc:729] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.025315 7973 consensus_queue.cc:260] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:55.025697 7973 ts_tablet_manager.cc:1398] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:55.030830 7969 tablet_bootstrap.cc:492] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:55.030984 7969 ts_tablet_manager.cc:1367] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:55.031456 7969 raft_consensus.cc:348] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:55.031914 7969 raft_consensus.cc:374] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.032012 7969 raft_consensus.cc:729] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.032196 7969 consensus_queue.cc:260] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:55.033519 7969 ts_tablet_manager.cc:1398] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:55.033728 7969 tablet_bootstrap.cc:492] T d6776209de5e405b982f863faa556727 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:55.035530 7969 tablet_bootstrap.cc:654] T d6776209de5e405b982f863faa556727 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.046052 7969 tablet_bootstrap.cc:492] T d6776209de5e405b982f863faa556727 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:55.046216 7969 ts_tablet_manager.cc:1367] T d6776209de5e405b982f863faa556727 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:09:55.047539 7969 raft_consensus.cc:348] T d6776209de5e405b982f863faa556727 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:55.047744 7969 raft_consensus.cc:374] T d6776209de5e405b982f863faa556727 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.047828 7969 raft_consensus.cc:729] T d6776209de5e405b982f863faa556727 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.048017 7969 consensus_queue.cc:260] T d6776209de5e405b982f863faa556727 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:55.048655 32533 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:55.049266 7969 ts_tablet_manager.cc:1398] T d6776209de5e405b982f863faa556727 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:55.049482 7969 tablet_bootstrap.cc:492] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:55.050529 7969 tablet_bootstrap.cc:654] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.051541 30257 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:55.056509 29994 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:55.059832 7382 consensus_queue.cc:1035] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:55.062930 6424 consensus_queue.cc:1035] T e6e4b617c50f4441bded13f9551362a0 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:55.068348 7969 tablet_bootstrap.cc:492] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:55.068447 7969 ts_tablet_manager.cc:1367] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 14:09:55.068904 7969 raft_consensus.cc:348] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.069041 7969 raft_consensus.cc:374] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.069082 7969 raft_consensus.cc:729] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.069190 7969 consensus_queue.cc:260] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.069551 7969 ts_tablet_manager.cc:1398] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:55.073611 327 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:55.075757 6683 raft_consensus.cc:482] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:55.075839 6683 raft_consensus.cc:504] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:55.076174 6683 leader_election.cc:288] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:55.076602 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b944c93f521418baf4c3a65635fbeca" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:55.076613 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b944c93f521418baf4c3a65635fbeca" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:55.076736 32553 raft_consensus.cc:2454] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:55.076738 32416 raft_consensus.cc:2454] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:55.076994 30212 leader_election.cc:302] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:55.077203 6683 raft_consensus.cc:2781] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:55.077261 6683 raft_consensus.cc:482] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:55.077311 6683 raft_consensus.cc:3037] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.078449 6683 raft_consensus.cc:504] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:55.078786 6683 leader_election.cc:288] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:55.079331 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b944c93f521418baf4c3a65635fbeca" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:55.079427 32416 raft_consensus.cc:3037] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.080638 32416 raft_consensus.cc:2454] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:55.080927 30212 leader_election.cc:302] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:55.081133 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b944c93f521418baf4c3a65635fbeca" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:55.081270 32553 raft_consensus.cc:3037] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.082257 32553 raft_consensus.cc:2454] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:55.082686 6683 raft_consensus.cc:2781] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:55.082790 30388 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:55.082875 6683 raft_consensus.cc:686] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:55.082998 6683 consensus_queue.cc:237] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:55.085237 29873 catalog_manager.cc:5484] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:55.100539 30125 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:55.108472 7963 tablet_bootstrap.cc:492] T 3125c67fe40641c2bfd5dcac81e7237f P db935501e9104de5b15c8780681d6c38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:55.108785 7963 tablet_bootstrap.cc:492] T 3125c67fe40641c2bfd5dcac81e7237f P db935501e9104de5b15c8780681d6c38: Bootstrap complete.
I20240501 14:09:55.108950 7963 ts_tablet_manager.cc:1367] T 3125c67fe40641c2bfd5dcac81e7237f P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.115s user 0.022s sys 0.003s
I20240501 14:09:55.109355 7963 raft_consensus.cc:348] T 3125c67fe40641c2bfd5dcac81e7237f P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:55.109479 7963 raft_consensus.cc:729] T 3125c67fe40641c2bfd5dcac81e7237f P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.109591 7963 consensus_queue.cc:260] T 3125c67fe40641c2bfd5dcac81e7237f P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:55.109850 7963 ts_tablet_manager.cc:1398] T 3125c67fe40641c2bfd5dcac81e7237f P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:55.110706 30165 tablet_copy_service.cc:342] P d35c16dcb506426993811bb2519d7db3: Request end of tablet copy session db935501e9104de5b15c8780681d6c38-3125c67fe40641c2bfd5dcac81e7237f received from {username='slave'} at 127.22.252.135:49787
I20240501 14:09:55.110791 30165 tablet_copy_service.cc:433] P d35c16dcb506426993811bb2519d7db3: ending tablet copy session db935501e9104de5b15c8780681d6c38-3125c67fe40641c2bfd5dcac81e7237f on tablet 3125c67fe40641c2bfd5dcac81e7237f with peer db935501e9104de5b15c8780681d6c38
I20240501 14:09:55.116149 32665 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:55.120894 8005 raft_consensus.cc:482] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:55.120993 8005 raft_consensus.cc:504] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.121405 8005 leader_election.cc:288] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:55.122296 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4462d7eaaa54223998369121761acd6" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:55.122423 32553 raft_consensus.cc:2454] T a4462d7eaaa54223998369121761acd6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:55.122723 30213 leader_election.cc:302] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 5f8a0cfdfdca4148a75686dc88059cf3; no voters:
I20240501 14:09:55.124261 8005 raft_consensus.cc:2781] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:55.124320 8005 raft_consensus.cc:482] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:55.124356 8005 raft_consensus.cc:3037] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.124347 32400 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:55.125475 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4462d7eaaa54223998369121761acd6" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:55.125555 8005 raft_consensus.cc:504] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.125609 349 raft_consensus.cc:2454] T a4462d7eaaa54223998369121761acd6 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:55.137912 8005 leader_election.cc:288] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:55.138440 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4462d7eaaa54223998369121761acd6" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:55.138581 32553 raft_consensus.cc:3037] T a4462d7eaaa54223998369121761acd6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.139568 32553 raft_consensus.cc:2454] T a4462d7eaaa54223998369121761acd6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:55.139871 30213 leader_election.cc:302] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 5f8a0cfdfdca4148a75686dc88059cf3; no voters:
I20240501 14:09:55.139892 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4462d7eaaa54223998369121761acd6" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:55.140014 349 raft_consensus.cc:3037] T a4462d7eaaa54223998369121761acd6 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.140039 8005 raft_consensus.cc:2781] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:55.140108 8005 raft_consensus.cc:686] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:55.140224 8005 consensus_queue.cc:237] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.141009 349 raft_consensus.cc:2454] T a4462d7eaaa54223998369121761acd6 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:55.144970 29873 catalog_manager.cc:5484] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:55.179402 30145 consensus_queue.cc:237] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:55.180691 30406 raft_consensus.cc:1261] T f740d894942a4bb584ca5688ef823d55 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:55.180692 32683 raft_consensus.cc:1261] T f740d894942a4bb584ca5688ef823d55 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:55.181147 7205 consensus_queue.cc:1035] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:55.181465 7561 consensus_queue.cc:1035] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:55.183105 7205 raft_consensus.cc:2932] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:55.183530 30406 raft_consensus.cc:2932] T f740d894942a4bb584ca5688ef823d55 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:55.184501 32683 raft_consensus.cc:2932] T f740d894942a4bb584ca5688ef823d55 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:55.186400 29873 catalog_manager.cc:5484] T f740d894942a4bb584ca5688ef823d55 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:55.190155 30257 tablet_service.cc:1514] Processing DeleteTablet for tablet f740d894942a4bb584ca5688ef823d55 with delete_type TABLET_DATA_TOMBSTONED (TS 1d0bb69611af4876a2b3b33c4766e13a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:55.191370 32553 raft_consensus.cc:1203] T cfc701c19cd74d0db2f90ce580230177 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:55.196463 8026 tablet_replica.cc:330] T f740d894942a4bb584ca5688ef823d55 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:55.196616 8026 raft_consensus.cc:2229] T f740d894942a4bb584ca5688ef823d55 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:55.196784 8026 raft_consensus.cc:2258] T f740d894942a4bb584ca5688ef823d55 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:55.197291 8026 ts_tablet_manager.cc:1875] T f740d894942a4bb584ca5688ef823d55 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:55.199498 8026 ts_tablet_manager.cc:1888] T f740d894942a4bb584ca5688ef823d55 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:55.199569 8026 log.cc:1196] T f740d894942a4bb584ca5688ef823d55 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/f740d894942a4bb584ca5688ef823d55
I20240501 14:09:55.200098 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet f740d894942a4bb584ca5688ef823d55 (table rebalance_test_table_1 [id=df055757a36d4b9f89886608b024f0b8]) successfully deleted
I20240501 14:09:55.204368 29868 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:55.208153 32416 raft_consensus.cc:1261] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:55.208173 32553 raft_consensus.cc:1261] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:55.208721 6683 consensus_queue.cc:1035] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:55.208925 8005 consensus_queue.cc:1035] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:55.211962 29868 catalog_manager.cc:5576] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265) does not have the latest schema for tablet 6b944c93f521418baf4c3a65635fbeca (table rebalancer_dynamic_table [id=f14fb343f1104929b44c1fbcfb443ad5]). Expected version 1 got 0
I20240501 14:09:55.213441 7840 tablet.cc:1722] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:55.213732 7840 tablet.cc:1620] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:55.214591 29870 catalog_manager.cc:5576] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497) does not have the latest schema for tablet 6b944c93f521418baf4c3a65635fbeca (table rebalancer_dynamic_table [id=f14fb343f1104929b44c1fbcfb443ad5]). Expected version 1 got 0
I20240501 14:09:55.214906 7841 tablet.cc:1722] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:55.215190 7841 tablet.cc:1620] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:55.217382 8040 tablet.cc:1722] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:55.217697 8040 tablet.cc:1620] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a: MemRowSet was empty: no flush needed.
I20240501 14:09:55.218559 29870 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=f14fb343f1104929b44c1fbcfb443ad5] alter complete (version 1)
I20240501 14:09:55.219374 7999 raft_consensus.cc:482] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:55.219463 7999 raft_consensus.cc:504] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:55.219921 7999 leader_election.cc:288] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:55.220826 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6776209de5e405b982f863faa556727" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:55.220943 30264 raft_consensus.cc:2454] T d6776209de5e405b982f863faa556727 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:55.221860 32358 leader_election.cc:302] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:55.222250 7999 raft_consensus.cc:2781] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:55.222306 7999 raft_consensus.cc:482] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:55.222344 7999 raft_consensus.cc:3037] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.223331 7999 raft_consensus.cc:504] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:55.223650 7999 leader_election.cc:288] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:55.223898 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6776209de5e405b982f863faa556727" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:55.223958 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6776209de5e405b982f863faa556727" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:55.224009 32554 raft_consensus.cc:3037] T d6776209de5e405b982f863faa556727 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.224047 30264 raft_consensus.cc:3037] T d6776209de5e405b982f863faa556727 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.224082 32553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6776209de5e405b982f863faa556727" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:55.224897 32554 raft_consensus.cc:2454] T d6776209de5e405b982f863faa556727 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:55.225005 30264 raft_consensus.cc:2454] T d6776209de5e405b982f863faa556727 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:55.226125 32358 leader_election.cc:302] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:55.226621 7999 raft_consensus.cc:2781] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:55.226688 7999 raft_consensus.cc:686] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:55.226804 7999 consensus_queue.cc:237] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:55.229497 29868 catalog_manager.cc:5484] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:55.246351 7575 raft_consensus.cc:482] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:55.246474 7575 raft_consensus.cc:504] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:55.246970 7575 leader_election.cc:288] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:55.247272 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59b8b59d67c64e2d828aaaa93647a5a4" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:55.247309 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59b8b59d67c64e2d828aaaa93647a5a4" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:55.247428 30406 raft_consensus.cc:2454] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:55.247433 32415 raft_consensus.cc:2454] T 59b8b59d67c64e2d828aaaa93647a5a4 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:09:55.247722 32620 leader_election.cc:302] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:55.247922 7575 raft_consensus.cc:2781] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:55.248005 7575 raft_consensus.cc:482] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:55.248057 7575 raft_consensus.cc:3037] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.249788 7575 raft_consensus.cc:504] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:55.251758 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59b8b59d67c64e2d828aaaa93647a5a4" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:55.251886 32415 raft_consensus.cc:3037] T 59b8b59d67c64e2d828aaaa93647a5a4 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.252451 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59b8b59d67c64e2d828aaaa93647a5a4" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:55.252586 30406 raft_consensus.cc:3037] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.252909 32415 raft_consensus.cc:2454] T 59b8b59d67c64e2d828aaaa93647a5a4 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:55.253286 32620 leader_election.cc:302] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:55.253484 7979 raft_consensus.cc:2781] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:55.253573 7979 raft_consensus.cc:686] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Becoming Leader. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
I20240501 14:09:55.253695 30406 raft_consensus.cc:2454] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:09:55.253722 7979 consensus_queue.cc:237] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:55.253928 7575 leader_election.cc:288] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:55.256217 29868 catalog_manager.cc:5484] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db935501e9104de5b15c8780681d6c38 (127.22.252.135). New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:55.320012 29864 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:55.328934 8040 tablet.cc:1722] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:55.329301 8040 tablet.cc:1620] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a: MemRowSet was empty: no flush needed.
I20240501 14:09:55.332850 32533 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:55.335638 29994 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:55.335883 7840 tablet.cc:1722] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:55.336207 7840 tablet.cc:1620] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:55.339255 29864 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=f14fb343f1104929b44c1fbcfb443ad5] alter complete (version 2)
I20240501 14:09:55.339783 7841 tablet.cc:1722] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:55.340080 7841 tablet.cc:1620] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:55.346051 32400 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:55.348922 30388 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:55.351483 32665 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:55.355355 327 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:55.373524 30125 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:55.375052 30257 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:55.429564 32553 consensus_queue.cc:237] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:55.430680 30264 raft_consensus.cc:1261] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:55.430707 32685 raft_consensus.cc:1261] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:55.431043 7853 consensus_queue.cc:1035] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:55.431365 6929 consensus_queue.cc:1035] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:55.432792 6929 raft_consensus.cc:2932] T 21efaac3a3d643679c5abaaddb9e5a3a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:55.433121 30264 raft_consensus.cc:2932] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:55.435532 32685 raft_consensus.cc:2932] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:55.436700 29870 catalog_manager.cc:5484] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:55.442502 327 tablet_service.cc:1514] Processing DeleteTablet for tablet 21efaac3a3d643679c5abaaddb9e5a3a with delete_type TABLET_DATA_TOMBSTONED (TS 9ecd907cf5634ac98f3f54f3dbec6849 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:55.443663 8107 tablet_replica.cc:330] T 21efaac3a3d643679c5abaaddb9e5a3a P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:55.443809 8107 raft_consensus.cc:2229] T 21efaac3a3d643679c5abaaddb9e5a3a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:55.443933 8107 raft_consensus.cc:2258] T 21efaac3a3d643679c5abaaddb9e5a3a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:55.444948 29866 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:55.445492 8107 ts_tablet_manager.cc:1875] T 21efaac3a3d643679c5abaaddb9e5a3a P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:55.448300 8107 ts_tablet_manager.cc:1888] T 21efaac3a3d643679c5abaaddb9e5a3a P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:09:55.448698 8107 log.cc:1196] T 21efaac3a3d643679c5abaaddb9e5a3a P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/21efaac3a3d643679c5abaaddb9e5a3a
I20240501 14:09:55.449980 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 21efaac3a3d643679c5abaaddb9e5a3a (table rebalancer_extra_table_2 [id=e53010e4736c42a899edac137f9182cb]) successfully deleted
I20240501 14:09:55.450507 8040 tablet.cc:1722] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:55.451180 7841 tablet.cc:1722] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:55.451498 7840 tablet.cc:1722] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:55.453650 32685 raft_consensus.cc:1203] T 3125c67fe40641c2bfd5dcac81e7237f P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:55.456357 29864 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=f14fb343f1104929b44c1fbcfb443ad5] alter complete (version 3)
I20240501 14:09:55.541139 6683 consensus_queue.cc:1035] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:55.545938 8005 consensus_queue.cc:1035] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:55.558557 29870 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:55.558686 29870 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:55.560796 29870 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 6b944c93f521418baf4c3a65635fbeca
I20240501 14:09:55.561437 32533 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b944c93f521418baf4c3a65635fbeca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:55 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:55.561811 30257 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b944c93f521418baf4c3a65635fbeca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:55 UTC) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:55.561977 8026 tablet_replica.cc:330] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:55.561971 32400 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b944c93f521418baf4c3a65635fbeca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:55 UTC) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:55.562122 8026 raft_consensus.cc:2229] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:55.562332 8026 raft_consensus.cc:2258] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:55.562875 8168 tablet_replica.cc:330] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:55.562902 8026 ts_tablet_manager.cc:1875] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:55.563071 8168 raft_consensus.cc:2229] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:55.563189 8168 raft_consensus.cc:2258] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:55.563546 8167 tablet_replica.cc:330] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:55.563704 8167 raft_consensus.cc:2229] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:55.563846 8167 raft_consensus.cc:2258] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:55.564301 8168 ts_tablet_manager.cc:1875] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:55.565053 8026 ts_tablet_manager.cc:1888] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:55.565115 8026 log.cc:1196] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/6b944c93f521418baf4c3a65635fbeca
I20240501 14:09:55.565389 8026 ts_tablet_manager.cc:1909] T 6b944c93f521418baf4c3a65635fbeca P 1d0bb69611af4876a2b3b33c4766e13a: Deleting consensus metadata
I20240501 14:09:55.565975 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet 6b944c93f521418baf4c3a65635fbeca (table rebalancer_dynamic_table_new_name [id=f14fb343f1104929b44c1fbcfb443ad5]) successfully deleted
I20240501 14:09:55.566911 8168 ts_tablet_manager.cc:1888] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:55.567013 8168 log.cc:1196] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/6b944c93f521418baf4c3a65635fbeca
I20240501 14:09:55.567401 8168 ts_tablet_manager.cc:1909] T 6b944c93f521418baf4c3a65635fbeca P a91029b90bf9415a89a2f70736759ad3: Deleting consensus metadata
I20240501 14:09:55.568101 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 6b944c93f521418baf4c3a65635fbeca (table rebalancer_dynamic_table_new_name [id=f14fb343f1104929b44c1fbcfb443ad5]) successfully deleted
I20240501 14:09:55.570469 8167 ts_tablet_manager.cc:1875] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:55.572888 8167 ts_tablet_manager.cc:1888] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:55.573014 8167 log.cc:1196] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/6b944c93f521418baf4c3a65635fbeca
I20240501 14:09:55.573339 8167 ts_tablet_manager.cc:1909] T 6b944c93f521418baf4c3a65635fbeca P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:55.575338 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet 6b944c93f521418baf4c3a65635fbeca (table rebalancer_dynamic_table_new_name [id=f14fb343f1104929b44c1fbcfb443ad5]) successfully deleted
I20240501 14:09:55.584965 30257 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:55.592113 327 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:55.600039 32665 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:55.603524 30388 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:55.614118 32533 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:55.615778 30125 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:55.617522 29994 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:55.650615 32400 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:55.663368 29871 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:55.668939 30125 tablet_service.cc:1467] Processing CreateTablet for tablet 23827ccc93e54e649859642b53156f67 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d23a06501c134900a1e65614666e932e]), partition=
I20240501 14:09:55.669219 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23827ccc93e54e649859642b53156f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.669209 327 tablet_service.cc:1467] Processing CreateTablet for tablet 23827ccc93e54e649859642b53156f67 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d23a06501c134900a1e65614666e932e]), partition=
I20240501 14:09:55.669222 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 23827ccc93e54e649859642b53156f67 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d23a06501c134900a1e65614666e932e]), partition=
I20240501 14:09:55.669410 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23827ccc93e54e649859642b53156f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.669447 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23827ccc93e54e649859642b53156f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.676434 8181 tablet_bootstrap.cc:492] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:55.676596 8182 tablet_bootstrap.cc:492] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:55.677522 8182 tablet_bootstrap.cc:654] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.677556 8181 tablet_bootstrap.cc:654] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.678256 8180 tablet_bootstrap.cc:492] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:55.679087 8180 tablet_bootstrap.cc:654] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.680393 8180 tablet_bootstrap.cc:492] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:55.680507 8180 ts_tablet_manager.cc:1367] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:55.681006 8180 raft_consensus.cc:348] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.681152 8180 raft_consensus.cc:374] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.681210 8180 raft_consensus.cc:729] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.681453 8180 consensus_queue.cc:260] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.681766 8180 ts_tablet_manager.cc:1398] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:55.686489 8181 tablet_bootstrap.cc:492] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:55.686591 8181 ts_tablet_manager.cc:1367] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:55.686771 8182 tablet_bootstrap.cc:492] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:55.686911 8182 ts_tablet_manager.cc:1367] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:55.687117 8181 raft_consensus.cc:348] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.687216 8181 raft_consensus.cc:374] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.687242 8181 raft_consensus.cc:729] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.687427 8182 raft_consensus.cc:348] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.687584 8182 raft_consensus.cc:374] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.687641 8182 raft_consensus.cc:729] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.687598 8181 consensus_queue.cc:260] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.687839 8182 consensus_queue.cc:260] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.688050 8181 ts_tablet_manager.cc:1398] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:55.688187 8182 ts_tablet_manager.cc:1398] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:55.706751 30406 consensus_queue.cc:237] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:55.708047 30145 raft_consensus.cc:1261] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:55.708047 32554 raft_consensus.cc:1261] T cfc701c19cd74d0db2f90ce580230177 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:55.708484 6185 consensus_queue.cc:1035] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:55.708659 7655 consensus_queue.cc:1035] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:55.710393 7655 raft_consensus.cc:2932] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:55.710601 7305 raft_consensus.cc:482] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:55.710685 7305 raft_consensus.cc:504] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.710686 30145 raft_consensus.cc:2932] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:55.710976 7305 leader_election.cc:288] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:55.711218 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23827ccc93e54e649859642b53156f67" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:55.711344 30146 raft_consensus.cc:2454] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:55.711323 347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23827ccc93e54e649859642b53156f67" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:55.711429 347 raft_consensus.cc:2454] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:55.711622 32359 leader_election.cc:302] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:55.711815 7999 raft_consensus.cc:2781] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:55.711869 7999 raft_consensus.cc:482] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:55.711905 7999 raft_consensus.cc:3037] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.712836 7999 raft_consensus.cc:504] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.713162 7999 leader_election.cc:288] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:55.713848 29866 catalog_manager.cc:5484] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3 reported cstate change: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:55.713973 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23827ccc93e54e649859642b53156f67" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:55.714068 30145 raft_consensus.cc:3037] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.714040 7305 consensus_queue.cc:1035] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:55.714515 347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23827ccc93e54e649859642b53156f67" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:55.714614 347 raft_consensus.cc:3037] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.714921 30145 raft_consensus.cc:2454] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:55.715195 32359 leader_election.cc:302] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:55.715344 7999 raft_consensus.cc:2781] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:55.715415 7999 raft_consensus.cc:686] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:55.715528 7999 consensus_queue.cc:237] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.715649 347 raft_consensus.cc:2454] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:55.715747 32552 raft_consensus.cc:2932] T cfc701c19cd74d0db2f90ce580230177 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:55.719089 7999 consensus_queue.cc:1035] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:55.719152 29865 catalog_manager.cc:5484] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:55.722858 30257 tablet_service.cc:1514] Processing DeleteTablet for tablet cfc701c19cd74d0db2f90ce580230177 with delete_type TABLET_DATA_TOMBSTONED (TS 1d0bb69611af4876a2b3b33c4766e13a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:55.723080 8026 tablet_replica.cc:330] T cfc701c19cd74d0db2f90ce580230177 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:55.723227 8026 raft_consensus.cc:2229] T cfc701c19cd74d0db2f90ce580230177 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:55.723383 8026 raft_consensus.cc:2258] T cfc701c19cd74d0db2f90ce580230177 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:55.723913 8026 ts_tablet_manager.cc:1875] T cfc701c19cd74d0db2f90ce580230177 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:55.725919 8026 ts_tablet_manager.cc:1888] T cfc701c19cd74d0db2f90ce580230177 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:55.725978 8026 log.cc:1196] T cfc701c19cd74d0db2f90ce580230177 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/cfc701c19cd74d0db2f90ce580230177
I20240501 14:09:55.726945 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet cfc701c19cd74d0db2f90ce580230177 (table rebalance_test_table_1 [id=df055757a36d4b9f89886608b024f0b8]) successfully deleted
I20240501 14:09:55.762908 29871 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:55.770483 30257 tablet_service.cc:1467] Processing CreateTablet for tablet 740debed04984ed4abe9b0cdf32f1562 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=b2179656a9324d9f96ca0af825446742]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:55.770843 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 740debed04984ed4abe9b0cdf32f1562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.770936 30125 tablet_service.cc:1467] Processing CreateTablet for tablet 0a49d9d1b13046c3bc1f9c73a83a3c3f (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=b2179656a9324d9f96ca0af825446742]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:55.771059 29994 tablet_service.cc:1467] Processing CreateTablet for tablet 0a49d9d1b13046c3bc1f9c73a83a3c3f (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=b2179656a9324d9f96ca0af825446742]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:55.771216 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a49d9d1b13046c3bc1f9c73a83a3c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.771337 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a49d9d1b13046c3bc1f9c73a83a3c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.771396 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 0a49d9d1b13046c3bc1f9c73a83a3c3f (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=b2179656a9324d9f96ca0af825446742]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:55.771651 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a49d9d1b13046c3bc1f9c73a83a3c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.772495 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 740debed04984ed4abe9b0cdf32f1562 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=b2179656a9324d9f96ca0af825446742]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:55.772711 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 740debed04984ed4abe9b0cdf32f1562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.772670 30126 tablet_service.cc:1467] Processing CreateTablet for tablet 43ecbf8c6104474fbf5eb9315f52056a (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=b2179656a9324d9f96ca0af825446742]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:55.772889 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43ecbf8c6104474fbf5eb9315f52056a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.773699 8182 tablet_bootstrap.cc:492] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:55.774714 8182 tablet_bootstrap.cc:654] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.775122 32533 tablet_service.cc:1467] Processing CreateTablet for tablet 43ecbf8c6104474fbf5eb9315f52056a (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=b2179656a9324d9f96ca0af825446742]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:55.775411 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43ecbf8c6104474fbf5eb9315f52056a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.775606 327 tablet_service.cc:1467] Processing CreateTablet for tablet 740debed04984ed4abe9b0cdf32f1562 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=b2179656a9324d9f96ca0af825446742]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:55.775832 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 740debed04984ed4abe9b0cdf32f1562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.778590 8180 tablet_bootstrap.cc:492] T 740debed04984ed4abe9b0cdf32f1562 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:55.779557 8180 tablet_bootstrap.cc:654] T 740debed04984ed4abe9b0cdf32f1562 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.780371 8251 tablet_bootstrap.cc:492] T 43ecbf8c6104474fbf5eb9315f52056a P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:55.780845 8248 tablet_bootstrap.cc:492] T 740debed04984ed4abe9b0cdf32f1562 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:55.781347 8249 tablet_bootstrap.cc:492] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:55.781668 8248 tablet_bootstrap.cc:654] T 740debed04984ed4abe9b0cdf32f1562 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.782620 8251 tablet_bootstrap.cc:654] T 43ecbf8c6104474fbf5eb9315f52056a P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.783651 8249 tablet_bootstrap.cc:654] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.784637 8251 tablet_bootstrap.cc:492] T 43ecbf8c6104474fbf5eb9315f52056a P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:55.784641 8182 tablet_bootstrap.cc:492] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:55.784726 8251 ts_tablet_manager.cc:1367] T 43ecbf8c6104474fbf5eb9315f52056a P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:55.784732 8182 ts_tablet_manager.cc:1367] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:09:55.785166 8182 raft_consensus.cc:348] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:55.785142 8251 raft_consensus.cc:348] T 43ecbf8c6104474fbf5eb9315f52056a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:55.785276 8182 raft_consensus.cc:374] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.785275 8251 raft_consensus.cc:374] T 43ecbf8c6104474fbf5eb9315f52056a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.785312 8251 raft_consensus.cc:729] T 43ecbf8c6104474fbf5eb9315f52056a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.785312 8182 raft_consensus.cc:729] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.785436 8182 consensus_queue.cc:260] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:55.785483 8251 consensus_queue.cc:260] T 43ecbf8c6104474fbf5eb9315f52056a P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:55.785732 8182 ts_tablet_manager.cc:1398] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:55.785888 8182 tablet_bootstrap.cc:492] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:55.785997 8251 ts_tablet_manager.cc:1398] T 43ecbf8c6104474fbf5eb9315f52056a P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:55.786684 8182 tablet_bootstrap.cc:654] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.788457 7979 consensus_queue.cc:1035] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:55.789516 29994 tablet_service.cc:1467] Processing CreateTablet for tablet 43ecbf8c6104474fbf5eb9315f52056a (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=b2179656a9324d9f96ca0af825446742]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:55.789760 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43ecbf8c6104474fbf5eb9315f52056a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.791797 8249 tablet_bootstrap.cc:492] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:55.791931 8249 ts_tablet_manager.cc:1367] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:09:55.792522 8249 raft_consensus.cc:348] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.792693 8249 raft_consensus.cc:374] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.792819 8249 raft_consensus.cc:729] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.793021 8249 consensus_queue.cc:260] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.794689 8249 ts_tablet_manager.cc:1398] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:55.794903 8249 tablet_bootstrap.cc:492] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:55.796170 8249 tablet_bootstrap.cc:654] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.798254 8249 tablet_bootstrap.cc:492] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:55.798367 8249 ts_tablet_manager.cc:1367] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:55.798786 8249 raft_consensus.cc:348] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:55.798900 8249 raft_consensus.cc:374] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.798942 8249 raft_consensus.cc:729] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.799077 8249 consensus_queue.cc:260] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:55.799324 8180 tablet_bootstrap.cc:492] T 740debed04984ed4abe9b0cdf32f1562 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:55.799347 8249 ts_tablet_manager.cc:1398] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:55.799401 8180 ts_tablet_manager.cc:1367] T 740debed04984ed4abe9b0cdf32f1562 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240501 14:09:55.799531 8249 tablet_bootstrap.cc:492] T 43ecbf8c6104474fbf5eb9315f52056a P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:55.799850 8180 raft_consensus.cc:348] T 740debed04984ed4abe9b0cdf32f1562 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.799949 8180 raft_consensus.cc:374] T 740debed04984ed4abe9b0cdf32f1562 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.799986 8180 raft_consensus.cc:729] T 740debed04984ed4abe9b0cdf32f1562 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.800098 8180 consensus_queue.cc:260] T 740debed04984ed4abe9b0cdf32f1562 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.800405 8180 ts_tablet_manager.cc:1398] T 740debed04984ed4abe9b0cdf32f1562 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:55.800639 8249 tablet_bootstrap.cc:654] T 43ecbf8c6104474fbf5eb9315f52056a P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.802629 8249 tablet_bootstrap.cc:492] T 43ecbf8c6104474fbf5eb9315f52056a P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:55.802821 8249 ts_tablet_manager.cc:1367] T 43ecbf8c6104474fbf5eb9315f52056a P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:55.803299 8248 tablet_bootstrap.cc:492] T 740debed04984ed4abe9b0cdf32f1562 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:55.804107 8248 ts_tablet_manager.cc:1367] T 740debed04984ed4abe9b0cdf32f1562 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240501 14:09:55.805359 8248 raft_consensus.cc:348] T 740debed04984ed4abe9b0cdf32f1562 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.805960 8248 raft_consensus.cc:374] T 740debed04984ed4abe9b0cdf32f1562 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.806043 8248 raft_consensus.cc:729] T 740debed04984ed4abe9b0cdf32f1562 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.806190 8248 consensus_queue.cc:260] T 740debed04984ed4abe9b0cdf32f1562 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.807657 8248 ts_tablet_manager.cc:1398] T 740debed04984ed4abe9b0cdf32f1562 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:55.809098 8249 raft_consensus.cc:348] T 43ecbf8c6104474fbf5eb9315f52056a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:55.809463 8249 raft_consensus.cc:374] T 43ecbf8c6104474fbf5eb9315f52056a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.809513 8249 raft_consensus.cc:729] T 43ecbf8c6104474fbf5eb9315f52056a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.809660 8249 consensus_queue.cc:260] T 43ecbf8c6104474fbf5eb9315f52056a P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:55.810781 8249 ts_tablet_manager.cc:1398] T 43ecbf8c6104474fbf5eb9315f52056a P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:09:55.817243 7205 raft_consensus.cc:482] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:55.817340 7205 raft_consensus.cc:504] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:55.817749 7205 leader_election.cc:288] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:55.818217 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a49d9d1b13046c3bc1f9c73a83a3c3f" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:55.818337 30013 raft_consensus.cc:2454] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:55.818555 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a49d9d1b13046c3bc1f9c73a83a3c3f" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:55.818636 8182 tablet_bootstrap.cc:492] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:55.818707 8182 ts_tablet_manager.cc:1367] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.000s
I20240501 14:09:55.818915 30081 leader_election.cc:302] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
W20240501 14:09:55.818994 30079 leader_election.cc:341] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:55.819099 8182 raft_consensus.cc:348] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:55.819185 8182 raft_consensus.cc:374] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.819212 8182 raft_consensus.cc:729] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.819316 8182 consensus_queue.cc:260] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:55.819453 7561 raft_consensus.cc:2781] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:55.819527 7561 raft_consensus.cc:482] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:55.819572 7561 raft_consensus.cc:3037] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.819577 8182 ts_tablet_manager.cc:1398] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:55.820782 8258 tablet_bootstrap.cc:492] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:55.820832 7561 raft_consensus.cc:504] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:55.821164 7561 leader_election.cc:288] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:55.822718 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a49d9d1b13046c3bc1f9c73a83a3c3f" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:55.822808 30013 raft_consensus.cc:3037] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.823781 30013 raft_consensus.cc:2454] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:55.823968 6832 consensus_queue.cc:1035] T 59b8b59d67c64e2d828aaaa93647a5a4 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:55.824124 30081 leader_election.cc:302] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:55.824134 29873 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:55.824754 7561 raft_consensus.cc:2781] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:55.824831 7561 raft_consensus.cc:686] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:55.824944 7561 consensus_queue.cc:237] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:55.823375 30408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a49d9d1b13046c3bc1f9c73a83a3c3f" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
W20240501 14:09:55.830327 30079 leader_election.cc:341] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:55.830410 30145 raft_consensus.cc:1261] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:55.830449 347 raft_consensus.cc:1261] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:55.830885 8258 tablet_bootstrap.cc:654] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.831192 7305 consensus_queue.cc:1035] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:55.834117 7999 consensus_queue.cc:1035] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:55.835932 29872 catalog_manager.cc:5576] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937) does not have the latest schema for tablet 23827ccc93e54e649859642b53156f67 (table rebalancer_dynamic_table [id=d23a06501c134900a1e65614666e932e]). Expected version 1 got 0
I20240501 14:09:55.840950 7840 tablet.cc:1722] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:55.841245 7840 tablet.cc:1620] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:55.841547 8258 tablet_bootstrap.cc:492] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:55.841689 8258 ts_tablet_manager.cc:1367] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240501 14:09:55.843917 8258 raft_consensus.cc:348] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:55.844172 8258 raft_consensus.cc:374] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.844274 8258 raft_consensus.cc:729] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.844605 8258 consensus_queue.cc:260] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:55.846314 8258 ts_tablet_manager.cc:1398] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:55.847872 8283 tablet.cc:1722] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:55.848168 8283 tablet.cc:1620] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3: MemRowSet was empty: no flush needed.
I20240501 14:09:55.850282 29870 catalog_manager.cc:5576] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937) does not have the latest schema for tablet 23827ccc93e54e649859642b53156f67 (table rebalancer_dynamic_table [id=d23a06501c134900a1e65614666e932e]). Expected version 1 got 0
I20240501 14:09:55.850497 7378 raft_consensus.cc:482] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:55.850607 7378 raft_consensus.cc:504] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.851063 7378 leader_election.cc:288] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:55.851861 347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740debed04984ed4abe9b0cdf32f1562" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:55.851990 347 raft_consensus.cc:2454] T 740debed04984ed4abe9b0cdf32f1562 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:55.852320 29950 leader_election.cc:302] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:55.852816 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740debed04984ed4abe9b0cdf32f1562" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:55.852942 30264 raft_consensus.cc:2454] T 740debed04984ed4abe9b0cdf32f1562 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:55.852993 7378 raft_consensus.cc:2781] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:55.853042 7378 raft_consensus.cc:482] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:55.853075 7378 raft_consensus.cc:3037] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.853785 8284 tablet.cc:1722] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:55.854028 7378 raft_consensus.cc:504] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.854228 8284 tablet.cc:1620] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:55.854343 7378 leader_election.cc:288] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:55.854816 347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740debed04984ed4abe9b0cdf32f1562" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:55.854933 347 raft_consensus.cc:3037] T 740debed04984ed4abe9b0cdf32f1562 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.855805 347 raft_consensus.cc:2454] T 740debed04984ed4abe9b0cdf32f1562 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:55.856127 29950 leader_election.cc:302] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:55.856432 7378 raft_consensus.cc:2781] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:55.856568 7378 raft_consensus.cc:686] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:55.856617 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740debed04984ed4abe9b0cdf32f1562" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:55.856705 30264 raft_consensus.cc:3037] T 740debed04984ed4abe9b0cdf32f1562 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.856683 7378 consensus_queue.cc:237] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:55.857661 30264 raft_consensus.cc:2454] T 740debed04984ed4abe9b0cdf32f1562 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:55.860838 29870 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=d23a06501c134900a1e65614666e932e] alter complete (version 1)
I20240501 14:09:55.860847 29872 catalog_manager.cc:5484] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:55.865072 29872 catalog_manager.cc:5576] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457) does not have the latest schema for tablet 23827ccc93e54e649859642b53156f67 (table rebalancer_dynamic_table [id=d23a06501c134900a1e65614666e932e]). Expected version 1 got 0
I20240501 14:09:55.865187 29872 catalog_manager.cc:5484] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:55.918713 30257 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:55.926352 7205 raft_consensus.cc:482] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:55.926470 7205 raft_consensus.cc:504] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:55.926882 7205 leader_election.cc:288] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:55.927279 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43ecbf8c6104474fbf5eb9315f52056a" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:55.927410 30013 raft_consensus.cc:2454] T 43ecbf8c6104474fbf5eb9315f52056a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:55.927714 32552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43ecbf8c6104474fbf5eb9315f52056a" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:55.927733 30081 leader_election.cc:302] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:55.927834 32552 raft_consensus.cc:2454] T 43ecbf8c6104474fbf5eb9315f52056a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:55.927877 7561 raft_consensus.cc:2781] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:55.927932 7561 raft_consensus.cc:482] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:55.927968 7561 raft_consensus.cc:3037] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.928922 327 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:55.929108 7561 raft_consensus.cc:504] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:55.929466 7561 leader_election.cc:288] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:55.930351 29994 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:55.930555 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43ecbf8c6104474fbf5eb9315f52056a" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:55.930665 30013 raft_consensus.cc:3037] T 43ecbf8c6104474fbf5eb9315f52056a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.930740 32552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43ecbf8c6104474fbf5eb9315f52056a" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:55.930862 32552 raft_consensus.cc:3037] T 43ecbf8c6104474fbf5eb9315f52056a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.931746 32552 raft_consensus.cc:2454] T 43ecbf8c6104474fbf5eb9315f52056a P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:55.931758 30013 raft_consensus.cc:2454] T 43ecbf8c6104474fbf5eb9315f52056a P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:09:55.932292 30081 leader_election.cc:302] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:55.932564 30126 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:09:55.932622 7205 raft_consensus.cc:2781] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:55.932694 7205 raft_consensus.cc:686] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:09:55.932830 7205 consensus_queue.cc:237] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:55.937340 32533 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:55.938537 29866 catalog_manager.cc:5484] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:55.939250 32400 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:55.944079 30388 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:55.946053 32665 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:55.963757 29866 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:55.964187 7655 raft_consensus.cc:482] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:55.964290 7655 raft_consensus.cc:504] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:55.964656 7655 leader_election.cc:288] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:55.965703 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a49d9d1b13046c3bc1f9c73a83a3c3f" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:55.966018 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a49d9d1b13046c3bc1f9c73a83a3c3f" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:55.966182 30013 raft_consensus.cc:2379] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1c217469ae4478ebd49e4ed9f7352d7 in current term 1: Already voted for candidate d35c16dcb506426993811bb2519d7db3 in this term.
I20240501 14:09:55.968515 30344 leader_election.cc:302] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7; no voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3
I20240501 14:09:55.969385 7655 raft_consensus.cc:3037] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.970510 7655 raft_consensus.cc:2726] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:09:55.972076 8283 tablet.cc:1722] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:55.972316 7840 tablet.cc:1722] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:55.972429 8283 tablet.cc:1620] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3: MemRowSet was empty: no flush needed.
I20240501 14:09:55.972577 7840 tablet.cc:1620] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:55.980265 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=d23a06501c134900a1e65614666e932e] alter complete (version 2)
I20240501 14:09:55.985435 8284 tablet.cc:1722] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:55.985698 8284 tablet.cc:1620] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:56.044454 30145 consensus_queue.cc:237] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:56.045583 32683 raft_consensus.cc:1261] T 3125c67fe40641c2bfd5dcac81e7237f P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:56.045677 30408 raft_consensus.cc:1261] T 3125c67fe40641c2bfd5dcac81e7237f P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:56.046006 7561 consensus_queue.cc:1035] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:56.046203 8299 consensus_queue.cc:1035] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:56.047713 7561 raft_consensus.cc:2932] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:56.048171 30408 raft_consensus.cc:2932] T 3125c67fe40641c2bfd5dcac81e7237f P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:56.048213 32683 raft_consensus.cc:2932] T 3125c67fe40641c2bfd5dcac81e7237f P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:56.051863 29866 catalog_manager.cc:5484] T 3125c67fe40641c2bfd5dcac81e7237f P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index 4 to 5, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:56.054441 30257 tablet_service.cc:1514] Processing DeleteTablet for tablet 3125c67fe40641c2bfd5dcac81e7237f with delete_type TABLET_DATA_TOMBSTONED (TS 1d0bb69611af4876a2b3b33c4766e13a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:56.054657 8026 tablet_replica.cc:330] T 3125c67fe40641c2bfd5dcac81e7237f P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:56.054802 8026 raft_consensus.cc:2229] T 3125c67fe40641c2bfd5dcac81e7237f P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:56.054941 8026 raft_consensus.cc:2258] T 3125c67fe40641c2bfd5dcac81e7237f P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:56.055466 8026 ts_tablet_manager.cc:1875] T 3125c67fe40641c2bfd5dcac81e7237f P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:56.057407 8026 ts_tablet_manager.cc:1888] T 3125c67fe40641c2bfd5dcac81e7237f P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:09:56.057464 8026 log.cc:1196] T 3125c67fe40641c2bfd5dcac81e7237f P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/3125c67fe40641c2bfd5dcac81e7237f
I20240501 14:09:56.058063 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet 3125c67fe40641c2bfd5dcac81e7237f (table rebalance_test_table_9 [id=abd0f09508ae4b6699a526d040fd6f96]) successfully deleted
I20240501 14:09:56.084170 29868 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:56.089779 7840 tablet.cc:1722] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:56.090168 8284 tablet.cc:1722] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:56.090989 8283 tablet.cc:1722] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:56.096163 29868 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=d23a06501c134900a1e65614666e932e] alter complete (version 3)
I20240501 14:09:56.200001 29873 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:56.200151 29873 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:56.202648 29873 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 23827ccc93e54e649859642b53156f67
I20240501 14:09:56.203105 30126 tablet_service.cc:1514] Processing DeleteTablet for tablet 23827ccc93e54e649859642b53156f67 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:56 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:56.204319 32400 tablet_service.cc:1514] Processing DeleteTablet for tablet 23827ccc93e54e649859642b53156f67 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:56 UTC) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:56.205145 8375 tablet_replica.cc:330] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:56.205324 8375 raft_consensus.cc:2229] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:56.205454 8375 raft_consensus.cc:2258] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:56.205644 327 tablet_service.cc:1514] Processing DeleteTablet for tablet 23827ccc93e54e649859642b53156f67 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:56 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:56.206120 8375 ts_tablet_manager.cc:1875] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:56.208622 8375 ts_tablet_manager.cc:1888] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:56.208738 8375 log.cc:1196] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/23827ccc93e54e649859642b53156f67
I20240501 14:09:56.209121 8375 ts_tablet_manager.cc:1909] T 23827ccc93e54e649859642b53156f67 P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:56.209225 8376 tablet_replica.cc:330] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:56.209514 8376 raft_consensus.cc:2229] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:56.209750 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 23827ccc93e54e649859642b53156f67 (table rebalancer_dynamic_table_new_name [id=d23a06501c134900a1e65614666e932e]) successfully deleted
I20240501 14:09:56.209951 8376 raft_consensus.cc:2258] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:56.210290 8377 tablet_replica.cc:330] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:56.210464 8377 raft_consensus.cc:2229] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:56.210621 8377 raft_consensus.cc:2258] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:56.210636 8376 ts_tablet_manager.cc:1875] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:56.211642 8377 ts_tablet_manager.cc:1875] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:56.214387 8376 ts_tablet_manager.cc:1888] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:56.214488 8376 log.cc:1196] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/23827ccc93e54e649859642b53156f67
I20240501 14:09:56.214777 8376 ts_tablet_manager.cc:1909] T 23827ccc93e54e649859642b53156f67 P a91029b90bf9415a89a2f70736759ad3: Deleting consensus metadata
I20240501 14:09:56.214920 8377 ts_tablet_manager.cc:1888] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:56.215023 8377 log.cc:1196] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/23827ccc93e54e649859642b53156f67
I20240501 14:09:56.215343 8377 ts_tablet_manager.cc:1909] T 23827ccc93e54e649859642b53156f67 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:09:56.215622 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 23827ccc93e54e649859642b53156f67 (table rebalancer_dynamic_table_new_name [id=d23a06501c134900a1e65614666e932e]) successfully deleted
I20240501 14:09:56.216073 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 23827ccc93e54e649859642b53156f67 (table rebalancer_dynamic_table_new_name [id=d23a06501c134900a1e65614666e932e]) successfully deleted
I20240501 14:09:56.248330 29994 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:56.251919 32533 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:56.252422 7378 consensus_queue.cc:1035] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:56.261914 32400 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:56.262544 30257 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:56.265035 32665 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:56.267011 30126 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:09:56.274355 30388 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:56.276279 327 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:56.278971 8285 consensus_queue.cc:1035] T 740debed04984ed4abe9b0cdf32f1562 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:56.305518 29871 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:56.315408 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 2b79389f70a8422a83bd1dbb11c15205 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c78114d065fb416f90340edb2a2d6bb8]), partition=
I20240501 14:09:56.315527 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 2b79389f70a8422a83bd1dbb11c15205 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c78114d065fb416f90340edb2a2d6bb8]), partition=
I20240501 14:09:56.315696 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b79389f70a8422a83bd1dbb11c15205. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.315768 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b79389f70a8422a83bd1dbb11c15205. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.318111 8393 tablet_bootstrap.cc:492] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:56.318157 30257 tablet_service.cc:1467] Processing CreateTablet for tablet 2b79389f70a8422a83bd1dbb11c15205 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c78114d065fb416f90340edb2a2d6bb8]), partition=
I20240501 14:09:56.318419 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b79389f70a8422a83bd1dbb11c15205. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.319248 8393 tablet_bootstrap.cc:654] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.320214 8393 tablet_bootstrap.cc:492] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:56.320364 8393 ts_tablet_manager.cc:1367] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:56.322937 8396 tablet_bootstrap.cc:492] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:56.324398 8396 tablet_bootstrap.cc:654] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.325481 8396 tablet_bootstrap.cc:492] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:56.325557 8396 ts_tablet_manager.cc:1367] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:09:56.325948 8396 raft_consensus.cc:348] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:56.326082 8396 raft_consensus.cc:374] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.326110 8396 raft_consensus.cc:729] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.326215 8396 consensus_queue.cc:260] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:56.326462 8396 ts_tablet_manager.cc:1398] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:56.329377 8393 raft_consensus.cc:348] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:56.329596 8394 tablet_bootstrap.cc:492] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:56.330911 8393 raft_consensus.cc:374] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.331022 8393 raft_consensus.cc:729] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.331228 8393 consensus_queue.cc:260] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:56.331782 8393 ts_tablet_manager.cc:1398] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.011s user 0.000s sys 0.001s
I20240501 14:09:56.333423 7205 consensus_queue.cc:1035] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:56.333788 8394 tablet_bootstrap.cc:654] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.338030 7205 consensus_queue.cc:1035] T 0a49d9d1b13046c3bc1f9c73a83a3c3f P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:56.364966 8394 tablet_bootstrap.cc:492] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:56.365087 8394 ts_tablet_manager.cc:1367] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.000s
I20240501 14:09:56.365628 8394 raft_consensus.cc:348] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:56.365742 8394 raft_consensus.cc:374] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.365765 8394 raft_consensus.cc:729] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.365887 8394 consensus_queue.cc:260] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:56.366127 8394 ts_tablet_manager.cc:1398] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:56.399552 30014 consensus_queue.cc:237] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:56.401033 32683 raft_consensus.cc:1261] T 14267ff763f5409998f457e14e407ed0 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:56.401062 32416 raft_consensus.cc:1261] T 14267ff763f5409998f457e14e407ed0 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:56.401108 29948 consensus_peers.cc:485] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e -> Peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): Couldn't send request to peer d1c217469ae4478ebd49e4ed9f7352d7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14267ff763f5409998f457e14e407ed0. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:56.401445 8419 consensus_queue.cc:1035] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:56.401624 8285 consensus_queue.cc:1035] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:56.404517 8419 raft_consensus.cc:2932] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:56.404894 32683 raft_consensus.cc:2932] T 14267ff763f5409998f457e14e407ed0 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:56.406397 32416 raft_consensus.cc:2932] T 14267ff763f5409998f457e14e407ed0 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:56.407621 29871 catalog_manager.cc:5484] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:56.431015 30408 consensus_queue.cc:237] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:56.432432 30264 raft_consensus.cc:1261] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:56.432749 32683 raft_consensus.cc:1261] T dd2fd47ec6684cb28d2f4678d2e90e75 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 14:09:56.432950 30343 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): Couldn't send request to peer 9ecd907cf5634ac98f3f54f3dbec6849. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd2fd47ec6684cb28d2f4678d2e90e75. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:56.433017 7655 consensus_queue.cc:1035] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:56.433168 6185 consensus_queue.cc:1035] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:56.436902 6185 raft_consensus.cc:2932] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:56.437371 30264 raft_consensus.cc:2932] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:56.438995 32683 raft_consensus.cc:2932] T dd2fd47ec6684cb28d2f4678d2e90e75 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:56.443315 29873 catalog_manager.cc:5484] T dd2fd47ec6684cb28d2f4678d2e90e75 P db935501e9104de5b15c8780681d6c38 reported cstate change: config changed from index 5 to 6, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:56.444825 29871 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:56.451354 30257 tablet_service.cc:1467] Processing CreateTablet for tablet 60bae1191bee48dc846dab52864f700a (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7d050d5c096f4da3a852733f35a14283]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.451414 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 0b34bae2736747998f30b774f1d0d571 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7d050d5c096f4da3a852733f35a14283]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.451638 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60bae1191bee48dc846dab52864f700a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.451670 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b34bae2736747998f30b774f1d0d571. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.451742 329 tablet_service.cc:1467] Processing CreateTablet for tablet 838d383811344878812c688aa39df801 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7d050d5c096f4da3a852733f35a14283]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.451771 327 tablet_service.cc:1467] Processing CreateTablet for tablet 60bae1191bee48dc846dab52864f700a (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7d050d5c096f4da3a852733f35a14283]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.452040 8437 raft_consensus.cc:482] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:56.452113 8437 raft_consensus.cc:504] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:56.452553 8437 leader_election.cc:288] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:56.453279 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b79389f70a8422a83bd1dbb11c15205" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:56.453331 30387 tablet_service.cc:1467] Processing CreateTablet for tablet 0b34bae2736747998f30b774f1d0d571 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7d050d5c096f4da3a852733f35a14283]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.453392 32683 raft_consensus.cc:2454] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:56.453583 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b34bae2736747998f30b774f1d0d571. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.453711 30210 leader_election.cc:302] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:56.454023 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 838d383811344878812c688aa39df801. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.454243 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 0b34bae2736747998f30b774f1d0d571 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7d050d5c096f4da3a852733f35a14283]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.454320 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 60bae1191bee48dc846dab52864f700a (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7d050d5c096f4da3a852733f35a14283]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.454478 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b34bae2736747998f30b774f1d0d571. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.454519 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60bae1191bee48dc846dab52864f700a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.456130 7999 raft_consensus.cc:482] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:56.456213 7999 raft_consensus.cc:504] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:56.456590 7999 leader_election.cc:288] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:56.456722 8396 tablet_bootstrap.cc:492] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:56.456750 30386 tablet_service.cc:1467] Processing CreateTablet for tablet 838d383811344878812c688aa39df801 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7d050d5c096f4da3a852733f35a14283]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.456949 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60bae1191bee48dc846dab52864f700a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.456959 30386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 838d383811344878812c688aa39df801. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.458084 8396 tablet_bootstrap.cc:654] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.458675 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b79389f70a8422a83bd1dbb11c15205" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:56.458784 32683 raft_consensus.cc:2454] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:56.458957 8394 tablet_bootstrap.cc:492] T 0b34bae2736747998f30b774f1d0d571 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:56.458995 8393 tablet_bootstrap.cc:492] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:56.459055 32356 leader_election.cc:302] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:56.460049 8393 tablet_bootstrap.cc:654] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.460119 8394 tablet_bootstrap.cc:654] T 0b34bae2736747998f30b774f1d0d571 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.460858 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 838d383811344878812c688aa39df801 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=7d050d5c096f4da3a852733f35a14283]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.460995 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b79389f70a8422a83bd1dbb11c15205" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:56.461040 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b79389f70a8422a83bd1dbb11c15205" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:56.461094 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 838d383811344878812c688aa39df801. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.461109 30264 raft_consensus.cc:2454] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:56.461130 32416 raft_consensus.cc:2454] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:56.461465 7999 raft_consensus.cc:2781] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:56.461514 7999 raft_consensus.cc:482] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:56.461549 7999 raft_consensus.cc:3037] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.462517 7999 raft_consensus.cc:504] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:56.462635 8396 tablet_bootstrap.cc:492] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:56.462710 8396 ts_tablet_manager.cc:1367] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 14:09:56.462888 7999 leader_election.cc:288] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:56.463080 8396 raft_consensus.cc:348] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:56.463182 8396 raft_consensus.cc:374] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.463207 8396 raft_consensus.cc:729] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.463316 8396 consensus_queue.cc:260] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:56.463596 8005 raft_consensus.cc:2781] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:56.463650 8005 raft_consensus.cc:482] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:56.463680 8005 raft_consensus.cc:3037] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.463771 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b79389f70a8422a83bd1dbb11c15205" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:56.463871 32685 raft_consensus.cc:3037] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.464068 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b79389f70a8422a83bd1dbb11c15205" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:56.464756 8005 raft_consensus.cc:504] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:56.464819 8394 tablet_bootstrap.cc:492] T 0b34bae2736747998f30b774f1d0d571 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:56.464865 32685 raft_consensus.cc:2454] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:56.464890 8394 ts_tablet_manager.cc:1367] T 0b34bae2736747998f30b774f1d0d571 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:56.465093 8005 leader_election.cc:288] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:56.465348 32356 leader_election.cc:302] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:56.465330 8394 raft_consensus.cc:348] T 0b34bae2736747998f30b774f1d0d571 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:56.465425 8394 raft_consensus.cc:374] T 0b34bae2736747998f30b774f1d0d571 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.465468 8394 raft_consensus.cc:729] T 0b34bae2736747998f30b774f1d0d571 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.465586 8394 consensus_queue.cc:260] T 0b34bae2736747998f30b774f1d0d571 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:56.465607 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b79389f70a8422a83bd1dbb11c15205" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:56.465727 32415 raft_consensus.cc:2379] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1d0bb69611af4876a2b3b33c4766e13a in current term 1: Already voted for candidate a91029b90bf9415a89a2f70736759ad3 in this term.
I20240501 14:09:56.465865 8394 ts_tablet_manager.cc:1398] T 0b34bae2736747998f30b774f1d0d571 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:56.466192 8448 tablet_bootstrap.cc:492] T 0b34bae2736747998f30b774f1d0d571 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:56.466208 7999 raft_consensus.cc:2781] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:56.466284 7999 raft_consensus.cc:686] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:56.466387 7999 consensus_queue.cc:237] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:56.467162 8448 tablet_bootstrap.cc:654] T 0b34bae2736747998f30b774f1d0d571 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.467543 30264 raft_consensus.cc:2379] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a91029b90bf9415a89a2f70736759ad3 in current term 1: Already voted for candidate 1d0bb69611af4876a2b3b33c4766e13a in this term.
I20240501 14:09:56.467905 8449 tablet_bootstrap.cc:492] T 838d383811344878812c688aa39df801 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:56.468884 8449 tablet_bootstrap.cc:654] T 838d383811344878812c688aa39df801 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.469017 29866 catalog_manager.cc:5484] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:56.470072 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b79389f70a8422a83bd1dbb11c15205" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:56.470189 32685 raft_consensus.cc:2379] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1d0bb69611af4876a2b3b33c4766e13a in current term 1: Already voted for candidate a91029b90bf9415a89a2f70736759ad3 in this term.
I20240501 14:09:56.470988 30210 leader_election.cc:302] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: a91029b90bf9415a89a2f70736759ad3, db935501e9104de5b15c8780681d6c38
I20240501 14:09:56.471244 8005 raft_consensus.cc:2726] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 14:09:56.471550 8396 ts_tablet_manager.cc:1398] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20240501 14:09:56.472604 8393 tablet_bootstrap.cc:492] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:56.472692 8393 ts_tablet_manager.cc:1367] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240501 14:09:56.473102 8393 raft_consensus.cc:348] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:56.473172 8393 raft_consensus.cc:374] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.473191 8393 raft_consensus.cc:729] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.473278 8393 consensus_queue.cc:260] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:56.473516 8393 ts_tablet_manager.cc:1398] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:56.473685 8393 tablet_bootstrap.cc:492] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:56.474846 8393 tablet_bootstrap.cc:654] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.475136 8448 tablet_bootstrap.cc:492] T 0b34bae2736747998f30b774f1d0d571 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:56.475351 8448 ts_tablet_manager.cc:1367] T 0b34bae2736747998f30b774f1d0d571 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:56.475853 8448 raft_consensus.cc:348] T 0b34bae2736747998f30b774f1d0d571 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:56.476023 8448 raft_consensus.cc:374] T 0b34bae2736747998f30b774f1d0d571 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.476109 8448 raft_consensus.cc:729] T 0b34bae2736747998f30b774f1d0d571 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.476317 8448 consensus_queue.cc:260] T 0b34bae2736747998f30b774f1d0d571 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:56.476616 8393 tablet_bootstrap.cc:492] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:56.476712 8393 ts_tablet_manager.cc:1367] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:09:56.476807 8448 ts_tablet_manager.cc:1398] T 0b34bae2736747998f30b774f1d0d571 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:56.476825 7205 consensus_queue.cc:1035] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:56.477211 8448 tablet_bootstrap.cc:492] T 60bae1191bee48dc846dab52864f700a P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:56.477195 8393 raft_consensus.cc:348] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:56.477281 8393 raft_consensus.cc:374] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.477315 8393 raft_consensus.cc:729] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.477428 8393 consensus_queue.cc:260] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:56.477692 8393 ts_tablet_manager.cc:1398] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:56.478106 8448 tablet_bootstrap.cc:654] T 60bae1191bee48dc846dab52864f700a P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.478173 8449 tablet_bootstrap.cc:492] T 838d383811344878812c688aa39df801 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:56.478251 8449 ts_tablet_manager.cc:1367] T 838d383811344878812c688aa39df801 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:56.478821 8449 raft_consensus.cc:348] T 838d383811344878812c688aa39df801 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:56.478899 8449 raft_consensus.cc:374] T 838d383811344878812c688aa39df801 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.478917 8449 raft_consensus.cc:729] T 838d383811344878812c688aa39df801 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.479007 8449 consensus_queue.cc:260] T 838d383811344878812c688aa39df801 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:56.479261 8449 ts_tablet_manager.cc:1398] T 838d383811344878812c688aa39df801 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:56.479447 8449 tablet_bootstrap.cc:492] T 60bae1191bee48dc846dab52864f700a P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:56.481447 8449 tablet_bootstrap.cc:654] T 60bae1191bee48dc846dab52864f700a P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.483788 7561 consensus_queue.cc:1035] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:56.485962 8448 tablet_bootstrap.cc:492] T 60bae1191bee48dc846dab52864f700a P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:56.486090 8448 ts_tablet_manager.cc:1367] T 60bae1191bee48dc846dab52864f700a P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:56.487794 8448 raft_consensus.cc:348] T 60bae1191bee48dc846dab52864f700a P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:56.487936 8448 raft_consensus.cc:374] T 60bae1191bee48dc846dab52864f700a P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.488005 8448 raft_consensus.cc:729] T 60bae1191bee48dc846dab52864f700a P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.488147 8448 consensus_queue.cc:260] T 60bae1191bee48dc846dab52864f700a P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:56.488517 8448 ts_tablet_manager.cc:1398] T 60bae1191bee48dc846dab52864f700a P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:56.488946 8448 tablet_bootstrap.cc:492] T 838d383811344878812c688aa39df801 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:56.489164 8449 tablet_bootstrap.cc:492] T 60bae1191bee48dc846dab52864f700a P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:56.489238 8449 ts_tablet_manager.cc:1367] T 60bae1191bee48dc846dab52864f700a P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240501 14:09:56.489651 8449 raft_consensus.cc:348] T 60bae1191bee48dc846dab52864f700a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:56.489744 8449 raft_consensus.cc:374] T 60bae1191bee48dc846dab52864f700a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.489782 8449 raft_consensus.cc:729] T 60bae1191bee48dc846dab52864f700a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.489851 8448 tablet_bootstrap.cc:654] T 838d383811344878812c688aa39df801 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.489925 8449 consensus_queue.cc:260] T 60bae1191bee48dc846dab52864f700a P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:56.490214 8449 ts_tablet_manager.cc:1398] T 60bae1191bee48dc846dab52864f700a P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:56.490900 8448 tablet_bootstrap.cc:492] T 838d383811344878812c688aa39df801 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:56.490968 8448 ts_tablet_manager.cc:1367] T 838d383811344878812c688aa39df801 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:56.491340 8448 raft_consensus.cc:348] T 838d383811344878812c688aa39df801 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:56.491428 8448 raft_consensus.cc:374] T 838d383811344878812c688aa39df801 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.491456 8448 raft_consensus.cc:729] T 838d383811344878812c688aa39df801 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.491549 8448 consensus_queue.cc:260] T 838d383811344878812c688aa39df801 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:56.491781 8448 ts_tablet_manager.cc:1398] T 838d383811344878812c688aa39df801 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:56.499790 30264 consensus_queue.cc:237] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:56.501061 349 raft_consensus.cc:1261] T a4462d7eaaa54223998369121761acd6 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:56.501492 32554 raft_consensus.cc:1261] T a4462d7eaaa54223998369121761acd6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:56.501543 30213 consensus_peers.cc:485] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Couldn't send request to peer d35c16dcb506426993811bb2519d7db3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4462d7eaaa54223998369121761acd6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:56.501994 8473 consensus_queue.cc:1035] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:56.502171 8005 consensus_queue.cc:1035] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:56.503715 8473 raft_consensus.cc:2932] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:56.504114 32554 raft_consensus.cc:2932] T a4462d7eaaa54223998369121761acd6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:56.507164 29866 catalog_manager.cc:5484] T a4462d7eaaa54223998369121761acd6 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:56.507781 349 raft_consensus.cc:2932] T a4462d7eaaa54223998369121761acd6 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:56.525481 30145 consensus_queue.cc:237] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:56.526742 32554 raft_consensus.cc:1261] T c7df6b8d76714eebbeacc85f403d3405 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:56.527052 32415 raft_consensus.cc:1261] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:56.527091 7561 consensus_queue.cc:1035] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:56.527292 30081 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): Couldn't send request to peer 93f2518fbadf4f9991cda93afacb363e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c7df6b8d76714eebbeacc85f403d3405. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:56.527540 7561 consensus_queue.cc:1035] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:56.528950 8005 raft_consensus.cc:482] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:56.529043 8005 raft_consensus.cc:504] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:56.529431 8005 leader_election.cc:288] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:56.529496 7561 raft_consensus.cc:2932] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:56.529676 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bae1191bee48dc846dab52864f700a" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:56.529783 349 raft_consensus.cc:2454] T 60bae1191bee48dc846dab52864f700a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:56.529780 30408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bae1191bee48dc846dab52864f700a" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:56.529883 30408 raft_consensus.cc:2454] T 60bae1191bee48dc846dab52864f700a P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:56.530041 30212 leader_election.cc:302] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:56.530180 8005 raft_consensus.cc:2781] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:56.530231 8005 raft_consensus.cc:482] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:56.530264 8005 raft_consensus.cc:3037] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.531121 8005 raft_consensus.cc:504] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:56.531404 8005 leader_election.cc:288] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:56.531524 32415 raft_consensus.cc:2932] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:56.531692 30408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bae1191bee48dc846dab52864f700a" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:56.531783 30408 raft_consensus.cc:3037] T 60bae1191bee48dc846dab52864f700a P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.532047 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bae1191bee48dc846dab52864f700a" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:56.531972 32554 raft_consensus.cc:2932] T c7df6b8d76714eebbeacc85f403d3405 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:56.532131 349 raft_consensus.cc:3037] T 60bae1191bee48dc846dab52864f700a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.532780 30408 raft_consensus.cc:2454] T 60bae1191bee48dc846dab52864f700a P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:56.533066 349 raft_consensus.cc:2454] T 60bae1191bee48dc846dab52864f700a P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:56.533365 30212 leader_election.cc:302] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:56.533552 8437 raft_consensus.cc:2781] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:56.533627 8437 raft_consensus.cc:686] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:56.533751 8437 consensus_queue.cc:237] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:56.535046 29873 catalog_manager.cc:5484] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index 3 to 4, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:56.539012 29871 catalog_manager.cc:5484] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:56.556486 32683 consensus_queue.cc:237] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:56.557700 32554 raft_consensus.cc:1261] T 8bb9687045d44efa888a69c7acde1105 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:56.557788 349 raft_consensus.cc:1261] T 8bb9687045d44efa888a69c7acde1105 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:56.558113 8511 consensus_queue.cc:1035] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:56.558382 8512 consensus_queue.cc:1035] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:09:56.558583 32618 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 -> Peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): Couldn't send request to peer d1c217469ae4478ebd49e4ed9f7352d7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8bb9687045d44efa888a69c7acde1105. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:56.560559 8511 raft_consensus.cc:2932] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:56.560941 349 raft_consensus.cc:2932] T 8bb9687045d44efa888a69c7acde1105 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:56.562500 32554 raft_consensus.cc:2932] T 8bb9687045d44efa888a69c7acde1105 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:56.563532 29871 catalog_manager.cc:5484] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 reported cstate change: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:56.580941 29866 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:56.582911 30408 consensus_queue.cc:237] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:56.584743 30264 raft_consensus.cc:1261] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:56.585088 7655 consensus_queue.cc:1035] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:09:56.585341 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c362bd01a59b4bcf85eeb59713e58b64. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:56.588637 32683 raft_consensus.cc:1261] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:56.589102 7655 consensus_queue.cc:1035] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:56.591205 7655 raft_consensus.cc:2932] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:56.592908 30264 raft_consensus.cc:2932] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:56.595211 29866 catalog_manager.cc:5484] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index 6 to 7, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New cstate: current_term: 2 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:56.597096 32683 raft_consensus.cc:2932] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:56.604758 32683 raft_consensus.cc:1261] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:56.604765 30264 raft_consensus.cc:1261] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:56.605131 7305 consensus_queue.cc:1035] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:56.605381 7305 consensus_queue.cc:1035] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:56.608232 29872 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet 2b79389f70a8422a83bd1dbb11c15205 (table rebalancer_dynamic_table [id=c78114d065fb416f90340edb2a2d6bb8]). Expected version 1 got 0
I20240501 14:09:56.609225 8547 tablet.cc:1722] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:56.609599 8547 tablet.cc:1620] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:56.610496 8546 tablet.cc:1722] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:56.610913 8546 tablet.cc:1620] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:56.611038 29872 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet 2b79389f70a8422a83bd1dbb11c15205 (table rebalancer_dynamic_table [id=c78114d065fb416f90340edb2a2d6bb8]). Expected version 1 got 0
I20240501 14:09:56.612553 29866 catalog_manager.cc:5576] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103) does not have the latest schema for tablet 2b79389f70a8422a83bd1dbb11c15205 (table rebalancer_dynamic_table [id=c78114d065fb416f90340edb2a2d6bb8]). Expected version 1 got 0
I20240501 14:09:56.618376 29866 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=c78114d065fb416f90340edb2a2d6bb8] alter complete (version 1)
I20240501 14:09:56.622114 8551 tablet.cc:1722] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:56.623768 29866 catalog_manager.cc:5576] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103) does not have the latest schema for tablet 2b79389f70a8422a83bd1dbb11c15205 (table rebalancer_dynamic_table [id=c78114d065fb416f90340edb2a2d6bb8]). Expected version 1 got 0
I20240501 14:09:56.625272 8551 tablet.cc:1620] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a: MemRowSet was empty: no flush needed.
I20240501 14:09:56.637709 30406 consensus_queue.cc:237] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:56.639062 32554 raft_consensus.cc:1261] T cfc701c19cd74d0db2f90ce580230177 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:56.639176 30145 raft_consensus.cc:1261] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:56.639573 8442 consensus_queue.cc:1035] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:09:56.639757 30343 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): Couldn't send request to peer 9ecd907cf5634ac98f3f54f3dbec6849. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cfc701c19cd74d0db2f90ce580230177. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:56.639927 8442 consensus_queue.cc:1035] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:56.641429 8442 raft_consensus.cc:2932] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:56.641901 30145 raft_consensus.cc:2932] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:56.645259 29872 catalog_manager.cc:5484] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index 5 to 6, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:56.645342 32554 raft_consensus.cc:2932] T cfc701c19cd74d0db2f90ce580230177 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:56.670902 349 consensus_queue.cc:237] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
W20240501 14:09:56.672366 32752 consensus_peers.cc:485] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): Couldn't send request to peer 93f2518fbadf4f9991cda93afacb363e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33de572fa35c40109cb70ee3287829f2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:56.672389 30145 raft_consensus.cc:1261] T 33de572fa35c40109cb70ee3287829f2 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:56.672811 32415 raft_consensus.cc:1261] T 33de572fa35c40109cb70ee3287829f2 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:56.672953 7382 consensus_queue.cc:1035] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:56.673132 6424 consensus_queue.cc:1035] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:56.675400 6424 raft_consensus.cc:2932] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:56.675850 30145 raft_consensus.cc:2932] T 33de572fa35c40109cb70ee3287829f2 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:56.677842 32415 raft_consensus.cc:2932] T 33de572fa35c40109cb70ee3287829f2 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:56.678609 29872 catalog_manager.cc:5484] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:56.683959 7305 raft_consensus.cc:482] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:56.684055 7305 raft_consensus.cc:504] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:56.684429 7305 leader_election.cc:288] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:56.684866 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838d383811344878812c688aa39df801" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:56.684913 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838d383811344878812c688aa39df801" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:56.684988 30406 raft_consensus.cc:2454] T 838d383811344878812c688aa39df801 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:56.685022 349 raft_consensus.cc:2454] T 838d383811344878812c688aa39df801 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:56.685249 32358 leader_election.cc:302] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:56.685429 7305 raft_consensus.cc:2781] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:56.685499 7305 raft_consensus.cc:482] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:56.685539 7305 raft_consensus.cc:3037] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.686264 7305 raft_consensus.cc:504] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:56.686524 7305 leader_election.cc:288] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:56.686925 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838d383811344878812c688aa39df801" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:56.686985 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838d383811344878812c688aa39df801" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:56.687032 349 raft_consensus.cc:3037] T 838d383811344878812c688aa39df801 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.687090 30406 raft_consensus.cc:3037] T 838d383811344878812c688aa39df801 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.687721 349 raft_consensus.cc:2454] T 838d383811344878812c688aa39df801 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:56.687773 30406 raft_consensus.cc:2454] T 838d383811344878812c688aa39df801 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:56.688022 32358 leader_election.cc:302] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:56.688220 7305 raft_consensus.cc:2781] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:56.688331 7305 raft_consensus.cc:686] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:56.688449 7305 consensus_queue.cc:237] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:56.690680 29866 catalog_manager.cc:5484] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:56.701473 349 consensus_queue.cc:237] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:56.702916 32415 raft_consensus.cc:1261] T 5db25d87e72347f4a4410bcde440f359 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:56.702934 32750 consensus_peers.cc:485] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): Couldn't send request to peer d1c217469ae4478ebd49e4ed9f7352d7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5db25d87e72347f4a4410bcde440f359. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:56.702960 32683 raft_consensus.cc:1261] T 5db25d87e72347f4a4410bcde440f359 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:56.703395 8573 consensus_queue.cc:1035] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:56.703596 7382 consensus_queue.cc:1035] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:56.705509 7382 raft_consensus.cc:2932] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:56.705808 32415 raft_consensus.cc:2932] T 5db25d87e72347f4a4410bcde440f359 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:56.708778 29873 catalog_manager.cc:5484] T 5db25d87e72347f4a4410bcde440f359 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:56.710549 32683 raft_consensus.cc:2932] T 5db25d87e72347f4a4410bcde440f359 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:56.723544 29866 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:56.729712 8546 tablet.cc:1722] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:56.730015 8546 tablet.cc:1620] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:56.730139 8547 tablet.cc:1722] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:56.730438 8547 tablet.cc:1620] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:09:56.730801 8551 tablet.cc:1722] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:56.731089 8551 tablet.cc:1620] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a: MemRowSet was empty: no flush needed.
I20240501 14:09:56.736318 29866 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=c78114d065fb416f90340edb2a2d6bb8] alter complete (version 2)
I20240501 14:09:56.738093 30406 consensus_queue.cc:237] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:56.739672 32415 raft_consensus.cc:1261] T b0ef35582c4f4d5c95199e00ccac92c9 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:56.739785 30343 consensus_peers.cc:485] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): Couldn't send request to peer 1d0bb69611af4876a2b3b33c4766e13a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0ef35582c4f4d5c95199e00ccac92c9. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:56.740072 7655 consensus_queue.cc:1035] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:56.740515 32554 raft_consensus.cc:1261] T b0ef35582c4f4d5c95199e00ccac92c9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:56.740886 7655 consensus_queue.cc:1035] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:56.742513 6185 raft_consensus.cc:2932] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:56.742924 32554 raft_consensus.cc:2932] T b0ef35582c4f4d5c95199e00ccac92c9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:56.745345 32415 raft_consensus.cc:2932] T b0ef35582c4f4d5c95199e00ccac92c9 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:56.746114 29871 catalog_manager.cc:5484] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:56.752764 7999 raft_consensus.cc:482] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:56.752856 7999 raft_consensus.cc:504] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:56.753230 7999 leader_election.cc:288] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:56.753618 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b34bae2736747998f30b774f1d0d571" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:56.753751 30406 raft_consensus.cc:2454] T 0b34bae2736747998f30b774f1d0d571 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:56.754076 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b34bae2736747998f30b774f1d0d571" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:56.754191 32685 raft_consensus.cc:2454] T 0b34bae2736747998f30b774f1d0d571 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:56.754382 32356 leader_election.cc:302] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:56.754554 7999 raft_consensus.cc:2781] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:56.754612 7999 raft_consensus.cc:482] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:56.754659 7999 raft_consensus.cc:3037] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.755596 7999 raft_consensus.cc:504] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:56.755897 7999 leader_election.cc:288] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:56.756196 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b34bae2736747998f30b774f1d0d571" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:56.756356 30406 raft_consensus.cc:3037] T 0b34bae2736747998f30b774f1d0d571 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.757246 30406 raft_consensus.cc:2454] T 0b34bae2736747998f30b774f1d0d571 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:56.757537 32356 leader_election.cc:302] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:56.757694 7999 raft_consensus.cc:2781] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:56.757681 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b34bae2736747998f30b774f1d0d571" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:56.757764 32685 raft_consensus.cc:3037] T 0b34bae2736747998f30b774f1d0d571 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.757779 7999 raft_consensus.cc:686] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:56.757900 7999 consensus_queue.cc:237] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:56.758728 32685 raft_consensus.cc:2454] T 0b34bae2736747998f30b774f1d0d571 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:56.760548 29866 catalog_manager.cc:5484] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:56.770696 30406 consensus_queue.cc:237] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:56.771939 349 raft_consensus.cc:1261] T 940c8f54339f4a32be2942682a41c685 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 14:09:56.772156 30343 consensus_peers.cc:485] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): Couldn't send request to peer a91029b90bf9415a89a2f70736759ad3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 940c8f54339f4a32be2942682a41c685. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:56.772380 8439 consensus_queue.cc:1035] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:56.772768 30145 raft_consensus.cc:1261] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:56.773150 8439 consensus_queue.cc:1035] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:56.775174 8442 raft_consensus.cc:2932] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:56.775745 30145 raft_consensus.cc:2932] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:56.777143 349 raft_consensus.cc:2932] T 940c8f54339f4a32be2942682a41c685 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:56.779213 29873 catalog_manager.cc:5484] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3 reported cstate change: config changed from index 5 to 6, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:56.798635 32685 consensus_queue.cc:237] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:56.800029 349 raft_consensus.cc:1261] T 6208e4b3d25941848b2ca533edd5a7b9 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:56.800083 30406 raft_consensus.cc:1261] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:56.800282 32620 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 -> Peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): Couldn't send request to peer 93f2518fbadf4f9991cda93afacb363e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6208e4b3d25941848b2ca533edd5a7b9. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:56.800572 8515 consensus_queue.cc:1035] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:56.801041 8511 consensus_queue.cc:1035] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:56.808967 8515 raft_consensus.cc:2932] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:56.809336 30406 raft_consensus.cc:2932] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:56.811965 349 raft_consensus.cc:2932] T 6208e4b3d25941848b2ca533edd5a7b9 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:56.812031 29866 catalog_manager.cc:5484] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 reported cstate change: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:56.837745 30014 consensus_queue.cc:237] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:56.839753 30406 raft_consensus.cc:1261] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:56.839833 347 raft_consensus.cc:1261] T af39c304693645269a6af59190e95cad P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:56.840162 8285 consensus_queue.cc:1035] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:56.840560 8285 consensus_queue.cc:1035] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:09:56.840735 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af39c304693645269a6af59190e95cad. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:56.841537 29873 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:56.842787 7378 raft_consensus.cc:2932] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:56.843133 347 raft_consensus.cc:2932] T af39c304693645269a6af59190e95cad P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:56.845608 30406 raft_consensus.cc:2932] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:56.846432 29872 catalog_manager.cc:5484] T af39c304693645269a6af59190e95cad P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: config changed from index 5 to 6, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:56.852838 8546 tablet.cc:1722] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:56.855279 8547 tablet.cc:1722] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:56.855759 8551 tablet.cc:1722] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:56.858646 29873 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=c78114d065fb416f90340edb2a2d6bb8] alter complete (version 3)
I20240501 14:09:56.875872 32683 consensus_queue.cc:237] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:56.877398 347 raft_consensus.cc:1261] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:56.877550 32620 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 -> Peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): Couldn't send request to peer 1d0bb69611af4876a2b3b33c4766e13a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e337eb723974834b07aac5eacf73232. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:56.877794 8511 consensus_queue.cc:1035] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:56.877825 32554 raft_consensus.cc:1261] T 7e337eb723974834b07aac5eacf73232 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:56.878268 8511 consensus_queue.cc:1035] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:56.880067 6832 raft_consensus.cc:2932] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:56.880455 32554 raft_consensus.cc:2932] T 7e337eb723974834b07aac5eacf73232 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:56.883478 29871 catalog_manager.cc:5484] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 reported cstate change: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:56.884455 349 raft_consensus.cc:2932] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:56.901897 30145 consensus_queue.cc:237] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:56.903468 30264 raft_consensus.cc:1261] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:56.903776 32554 raft_consensus.cc:1261] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:56.904116 8705 ts_tablet_manager.cc:902] T a4462d7eaaa54223998369121761acd6 P d35c16dcb506426993811bb2519d7db3: Initiating tablet copy from peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
W20240501 14:09:56.904219 30081 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 -> Peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): Couldn't send request to peer a91029b90bf9415a89a2f70736759ad3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5518112449d9448e98f5c779f1fa3248. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:56.904441 8493 consensus_queue.cc:1035] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:56.904668 8705 tablet_copy_client.cc:308] T a4462d7eaaa54223998369121761acd6 P d35c16dcb506426993811bb2519d7db3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.131:43103
I20240501 14:09:56.905014 8493 consensus_queue.cc:1035] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:56.905110 30297 tablet_copy_service.cc:140] P 1d0bb69611af4876a2b3b33c4766e13a: Received BeginTabletCopySession request for tablet a4462d7eaaa54223998369121761acd6 from peer d35c16dcb506426993811bb2519d7db3 ({username='slave'} at 127.22.252.130:58491)
I20240501 14:09:56.905194 30297 tablet_copy_service.cc:161] P 1d0bb69611af4876a2b3b33c4766e13a: Beginning new tablet copy session on tablet a4462d7eaaa54223998369121761acd6 from peer d35c16dcb506426993811bb2519d7db3 at {username='slave'} at 127.22.252.130:58491: session id = d35c16dcb506426993811bb2519d7db3-a4462d7eaaa54223998369121761acd6
I20240501 14:09:56.906142 30297 tablet_copy_source_session.cc:205] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:56.906694 8706 ts_tablet_manager.cc:902] T dd2fd47ec6684cb28d2f4678d2e90e75 P 9ecd907cf5634ac98f3f54f3dbec6849: Initiating tablet copy from peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:56.906728 8299 raft_consensus.cc:2932] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:56.907001 8705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4462d7eaaa54223998369121761acd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.907111 32554 raft_consensus.cc:2932] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:56.907368 8706 tablet_copy_client.cc:308] T dd2fd47ec6684cb28d2f4678d2e90e75 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.132:34357
I20240501 14:09:56.908939 30264 raft_consensus.cc:2932] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:56.909282 8705 tablet_copy_client.cc:783] T a4462d7eaaa54223998369121761acd6 P d35c16dcb506426993811bb2519d7db3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:56.909435 8705 tablet_copy_client.cc:647] T a4462d7eaaa54223998369121761acd6 P d35c16dcb506426993811bb2519d7db3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:56.909564 30428 tablet_copy_service.cc:140] P d1c217469ae4478ebd49e4ed9f7352d7: Received BeginTabletCopySession request for tablet dd2fd47ec6684cb28d2f4678d2e90e75 from peer 9ecd907cf5634ac98f3f54f3dbec6849 ({username='slave'} at 127.22.252.136:38643)
I20240501 14:09:56.909642 30428 tablet_copy_service.cc:161] P d1c217469ae4478ebd49e4ed9f7352d7: Beginning new tablet copy session on tablet dd2fd47ec6684cb28d2f4678d2e90e75 from peer 9ecd907cf5634ac98f3f54f3dbec6849 at {username='slave'} at 127.22.252.136:38643: session id = 9ecd907cf5634ac98f3f54f3dbec6849-dd2fd47ec6684cb28d2f4678d2e90e75
I20240501 14:09:56.910439 29873 catalog_manager.cc:5484] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 reported cstate change: config changed from index 5 to 6, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:56.910667 30428 tablet_copy_source_session.cc:205] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:56.913642 8705 tablet_copy_client.cc:523] T a4462d7eaaa54223998369121761acd6 P d35c16dcb506426993811bb2519d7db3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:56.914808 8705 tablet_bootstrap.cc:492] T a4462d7eaaa54223998369121761acd6 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:56.915407 8706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd2fd47ec6684cb28d2f4678d2e90e75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.917526 8706 tablet_copy_client.cc:783] T dd2fd47ec6684cb28d2f4678d2e90e75 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:56.917671 8706 tablet_copy_client.cc:647] T dd2fd47ec6684cb28d2f4678d2e90e75 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:56.918865 8706 tablet_copy_client.cc:523] T dd2fd47ec6684cb28d2f4678d2e90e75 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:56.919778 8705 tablet_bootstrap.cc:492] T a4462d7eaaa54223998369121761acd6 P d35c16dcb506426993811bb2519d7db3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:56.920266 8706 tablet_bootstrap.cc:492] T dd2fd47ec6684cb28d2f4678d2e90e75 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:56.920305 8705 tablet_bootstrap.cc:492] T a4462d7eaaa54223998369121761acd6 P d35c16dcb506426993811bb2519d7db3: Bootstrap complete.
I20240501 14:09:56.920485 8705 ts_tablet_manager.cc:1367] T a4462d7eaaa54223998369121761acd6 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 14:09:56.920929 8705 raft_consensus.cc:348] T a4462d7eaaa54223998369121761acd6 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:56.921044 8705 raft_consensus.cc:729] T a4462d7eaaa54223998369121761acd6 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.921164 8705 consensus_queue.cc:260] T a4462d7eaaa54223998369121761acd6 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:56.921480 8705 ts_tablet_manager.cc:1398] T a4462d7eaaa54223998369121761acd6 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:56.921916 30297 tablet_copy_service.cc:342] P 1d0bb69611af4876a2b3b33c4766e13a: Request end of tablet copy session d35c16dcb506426993811bb2519d7db3-a4462d7eaaa54223998369121761acd6 received from {username='slave'} at 127.22.252.130:58491
I20240501 14:09:56.921983 30297 tablet_copy_service.cc:433] P 1d0bb69611af4876a2b3b33c4766e13a: ending tablet copy session d35c16dcb506426993811bb2519d7db3-a4462d7eaaa54223998369121761acd6 on tablet a4462d7eaaa54223998369121761acd6 with peer d35c16dcb506426993811bb2519d7db3
I20240501 14:09:56.936574 32683 consensus_queue.cc:237] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:56.937907 32552 raft_consensus.cc:1261] T 0cccc8eec30141c1946b57d3be27000f P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:56.937959 347 raft_consensus.cc:1261] T 0cccc8eec30141c1946b57d3be27000f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:56.938316 8511 consensus_queue.cc:1035] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:09:56.938664 32620 consensus_peers.cc:485] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 -> Peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): Couldn't send request to peer 93f2518fbadf4f9991cda93afacb363e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0cccc8eec30141c1946b57d3be27000f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:56.938836 8511 consensus_queue.cc:1035] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:56.940572 8515 raft_consensus.cc:2932] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:56.941103 32554 raft_consensus.cc:2932] T 0cccc8eec30141c1946b57d3be27000f P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:56.942567 347 raft_consensus.cc:2932] T 0cccc8eec30141c1946b57d3be27000f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:56.944003 29871 catalog_manager.cc:5484] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 reported cstate change: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:56.947044 8706 tablet_bootstrap.cc:492] T dd2fd47ec6684cb28d2f4678d2e90e75 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:56.947358 8706 tablet_bootstrap.cc:492] T dd2fd47ec6684cb28d2f4678d2e90e75 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap complete.
I20240501 14:09:56.947544 8706 ts_tablet_manager.cc:1367] T dd2fd47ec6684cb28d2f4678d2e90e75 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.027s user 0.022s sys 0.000s
I20240501 14:09:56.947964 8706 raft_consensus.cc:348] T dd2fd47ec6684cb28d2f4678d2e90e75 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:56.948063 8706 raft_consensus.cc:729] T dd2fd47ec6684cb28d2f4678d2e90e75 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.948285 8706 consensus_queue.cc:260] T dd2fd47ec6684cb28d2f4678d2e90e75 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:56.948596 8706 ts_tablet_manager.cc:1398] T dd2fd47ec6684cb28d2f4678d2e90e75 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 14:09:56.949420 30428 tablet_copy_service.cc:342] P d1c217469ae4478ebd49e4ed9f7352d7: Request end of tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-dd2fd47ec6684cb28d2f4678d2e90e75 received from {username='slave'} at 127.22.252.136:38643
I20240501 14:09:56.949515 30428 tablet_copy_service.cc:433] P d1c217469ae4478ebd49e4ed9f7352d7: ending tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-dd2fd47ec6684cb28d2f4678d2e90e75 on tablet dd2fd47ec6684cb28d2f4678d2e90e75 with peer 9ecd907cf5634ac98f3f54f3dbec6849
I20240501 14:09:56.950357 8737 ts_tablet_manager.cc:902] T 14267ff763f5409998f457e14e407ed0 P d1c217469ae4478ebd49e4ed9f7352d7: Initiating tablet copy from peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:56.950874 8737 tablet_copy_client.cc:308] T 14267ff763f5409998f457e14e407ed0 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.129:41387
I20240501 14:09:56.951324 30035 tablet_copy_service.cc:140] P 93f2518fbadf4f9991cda93afacb363e: Received BeginTabletCopySession request for tablet 14267ff763f5409998f457e14e407ed0 from peer d1c217469ae4478ebd49e4ed9f7352d7 ({username='slave'} at 127.22.252.132:37497)
I20240501 14:09:56.951428 30035 tablet_copy_service.cc:161] P 93f2518fbadf4f9991cda93afacb363e: Beginning new tablet copy session on tablet 14267ff763f5409998f457e14e407ed0 from peer d1c217469ae4478ebd49e4ed9f7352d7 at {username='slave'} at 127.22.252.132:37497: session id = d1c217469ae4478ebd49e4ed9f7352d7-14267ff763f5409998f457e14e407ed0
I20240501 14:09:56.952385 30035 tablet_copy_source_session.cc:205] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:56.953186 8737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14267ff763f5409998f457e14e407ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.955276 8737 tablet_copy_client.cc:783] T 14267ff763f5409998f457e14e407ed0 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:56.955427 8737 tablet_copy_client.cc:647] T 14267ff763f5409998f457e14e407ed0 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:56.956560 8737 tablet_copy_client.cc:523] T 14267ff763f5409998f457e14e407ed0 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:56.957883 8737 tablet_bootstrap.cc:492] T 14267ff763f5409998f457e14e407ed0 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:56.961555 8737 tablet_bootstrap.cc:492] T 14267ff763f5409998f457e14e407ed0 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:56.961827 8737 tablet_bootstrap.cc:492] T 14267ff763f5409998f457e14e407ed0 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap complete.
I20240501 14:09:56.961942 29872 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:56.962069 29872 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:56.962276 8737 ts_tablet_manager.cc:1367] T 14267ff763f5409998f457e14e407ed0 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:09:56.962718 8737 raft_consensus.cc:348] T 14267ff763f5409998f457e14e407ed0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:56.962836 8737 raft_consensus.cc:729] T 14267ff763f5409998f457e14e407ed0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.962952 8737 consensus_queue.cc:260] T 14267ff763f5409998f457e14e407ed0 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:56.963236 8737 ts_tablet_manager.cc:1398] T 14267ff763f5409998f457e14e407ed0 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:56.963697 30035 tablet_copy_service.cc:342] P 93f2518fbadf4f9991cda93afacb363e: Request end of tablet copy session d1c217469ae4478ebd49e4ed9f7352d7-14267ff763f5409998f457e14e407ed0 received from {username='slave'} at 127.22.252.132:37497
I20240501 14:09:56.963773 30035 tablet_copy_service.cc:433] P 93f2518fbadf4f9991cda93afacb363e: ending tablet copy session d1c217469ae4478ebd49e4ed9f7352d7-14267ff763f5409998f457e14e407ed0 on tablet 14267ff763f5409998f457e14e407ed0 with peer d1c217469ae4478ebd49e4ed9f7352d7
I20240501 14:09:56.964372 29872 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 2b79389f70a8422a83bd1dbb11c15205
I20240501 14:09:56.964812 32665 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b79389f70a8422a83bd1dbb11c15205 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:56 UTC) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:56.965085 30257 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b79389f70a8422a83bd1dbb11c15205 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:56 UTC) from {username='slave'} at 127.0.0.1:44464
I20240501 14:09:56.965432 8754 tablet_replica.cc:330] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:09:56.965442 32400 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b79389f70a8422a83bd1dbb11c15205 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:56 UTC) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:56.965580 8754 raft_consensus.cc:2229] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:56.965710 8754 raft_consensus.cc:2258] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:56.966003 8755 tablet_replica.cc:330] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:56.966140 8755 raft_consensus.cc:2229] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:56.966329 8755 raft_consensus.cc:2258] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:56.966547 8753 tablet_replica.cc:330] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:56.966821 8754 ts_tablet_manager.cc:1875] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:56.968011 8755 ts_tablet_manager.cc:1875] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:56.968382 8753 raft_consensus.cc:2229] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:56.968566 8753 raft_consensus.cc:2258] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:56.969202 8753 ts_tablet_manager.cc:1875] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:56.969444 8754 ts_tablet_manager.cc:1888] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:56.969630 8754 log.cc:1196] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/2b79389f70a8422a83bd1dbb11c15205
I20240501 14:09:56.970060 8754 ts_tablet_manager.cc:1909] T 2b79389f70a8422a83bd1dbb11c15205 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting consensus metadata
I20240501 14:09:56.970448 8755 ts_tablet_manager.cc:1888] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:56.970574 8755 log.cc:1196] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/2b79389f70a8422a83bd1dbb11c15205
I20240501 14:09:56.970788 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet 2b79389f70a8422a83bd1dbb11c15205 (table rebalancer_dynamic_table_new_name [id=c78114d065fb416f90340edb2a2d6bb8]) successfully deleted
I20240501 14:09:56.970909 8755 ts_tablet_manager.cc:1909] T 2b79389f70a8422a83bd1dbb11c15205 P a91029b90bf9415a89a2f70736759ad3: Deleting consensus metadata
I20240501 14:09:56.971657 8753 ts_tablet_manager.cc:1888] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:56.971647 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 2b79389f70a8422a83bd1dbb11c15205 (table rebalancer_dynamic_table_new_name [id=c78114d065fb416f90340edb2a2d6bb8]) successfully deleted
I20240501 14:09:56.971753 8753 log.cc:1196] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/2b79389f70a8422a83bd1dbb11c15205
I20240501 14:09:56.972332 8753 ts_tablet_manager.cc:1909] T 2b79389f70a8422a83bd1dbb11c15205 P db935501e9104de5b15c8780681d6c38: Deleting consensus metadata
I20240501 14:09:56.973074 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet 2b79389f70a8422a83bd1dbb11c15205 (table rebalancer_dynamic_table_new_name [id=c78114d065fb416f90340edb2a2d6bb8]) successfully deleted
I20240501 14:09:56.981194 32683 consensus_queue.cc:237] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:56.982827 30406 raft_consensus.cc:1261] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:56.983208 30145 raft_consensus.cc:1261] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:56.983537 32620 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 -> Peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): Couldn't send request to peer 1d0bb69611af4876a2b3b33c4766e13a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8430bd8cb7549f0aa4ede9a3d1ffe6e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:56.983575 8515 consensus_queue.cc:1035] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:56.983706 8512 consensus_queue.cc:1035] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:56.994844 8512 raft_consensus.cc:2932] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:56.995170 30145 raft_consensus.cc:2932] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:56.996671 30406 raft_consensus.cc:2932] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:56.998327 29871 catalog_manager.cc:5484] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 reported cstate change: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:57.012360 8769 ts_tablet_manager.cc:902] T c7df6b8d76714eebbeacc85f403d3405 P 93f2518fbadf4f9991cda93afacb363e: Initiating tablet copy from peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:57.013178 8769 tablet_copy_client.cc:308] T c7df6b8d76714eebbeacc85f403d3405 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.130:36457
I20240501 14:09:57.014101 30165 tablet_copy_service.cc:140] P d35c16dcb506426993811bb2519d7db3: Received BeginTabletCopySession request for tablet c7df6b8d76714eebbeacc85f403d3405 from peer 93f2518fbadf4f9991cda93afacb363e ({username='slave'} at 127.22.252.129:42467)
I20240501 14:09:57.014189 30165 tablet_copy_service.cc:161] P d35c16dcb506426993811bb2519d7db3: Beginning new tablet copy session on tablet c7df6b8d76714eebbeacc85f403d3405 from peer 93f2518fbadf4f9991cda93afacb363e at {username='slave'} at 127.22.252.129:42467: session id = 93f2518fbadf4f9991cda93afacb363e-c7df6b8d76714eebbeacc85f403d3405
I20240501 14:09:57.015121 30165 tablet_copy_source_session.cc:205] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:57.016368 8769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7df6b8d76714eebbeacc85f403d3405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.018661 8769 tablet_copy_client.cc:783] T c7df6b8d76714eebbeacc85f403d3405 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:57.018832 8769 tablet_copy_client.cc:647] T c7df6b8d76714eebbeacc85f403d3405 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:57.021616 8769 tablet_copy_client.cc:523] T c7df6b8d76714eebbeacc85f403d3405 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:57.023136 8769 tablet_bootstrap.cc:492] T c7df6b8d76714eebbeacc85f403d3405 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:57.031142 8769 tablet_bootstrap.cc:492] T c7df6b8d76714eebbeacc85f403d3405 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:57.031415 8769 tablet_bootstrap.cc:492] T c7df6b8d76714eebbeacc85f403d3405 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap complete.
I20240501 14:09:57.031742 8769 ts_tablet_manager.cc:1367] T c7df6b8d76714eebbeacc85f403d3405 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20240501 14:09:57.032289 8769 raft_consensus.cc:348] T c7df6b8d76714eebbeacc85f403d3405 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:57.032486 8769 raft_consensus.cc:729] T c7df6b8d76714eebbeacc85f403d3405 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.032670 8769 consensus_queue.cc:260] T c7df6b8d76714eebbeacc85f403d3405 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:57.033068 8769 ts_tablet_manager.cc:1398] T c7df6b8d76714eebbeacc85f403d3405 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:57.033674 30165 tablet_copy_service.cc:342] P d35c16dcb506426993811bb2519d7db3: Request end of tablet copy session 93f2518fbadf4f9991cda93afacb363e-c7df6b8d76714eebbeacc85f403d3405 received from {username='slave'} at 127.22.252.129:42467
I20240501 14:09:57.033749 30165 tablet_copy_service.cc:433] P d35c16dcb506426993811bb2519d7db3: ending tablet copy session 93f2518fbadf4f9991cda93afacb363e-c7df6b8d76714eebbeacc85f403d3405 on tablet c7df6b8d76714eebbeacc85f403d3405 with peer 93f2518fbadf4f9991cda93afacb363e
I20240501 14:09:57.037034 347 consensus_queue.cc:237] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:57.038338 32683 raft_consensus.cc:1261] T bf0712a5e11c46a291b90bfb491b8b25 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:57.038431 32416 raft_consensus.cc:1261] T bf0712a5e11c46a291b90bfb491b8b25 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:57.038743 7382 consensus_queue.cc:1035] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:57.038895 8595 consensus_queue.cc:1035] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:09:57.040148 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf0712a5e11c46a291b90bfb491b8b25. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:57.041380 7382 raft_consensus.cc:2932] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:57.043179 32683 raft_consensus.cc:2932] T bf0712a5e11c46a291b90bfb491b8b25 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:57.043934 32416 raft_consensus.cc:2932] T bf0712a5e11c46a291b90bfb491b8b25 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:57.044528 29873 catalog_manager.cc:5484] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: config changed from index -1 to 2, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:57.056607 8437 consensus_queue.cc:1035] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:57.059331 8437 consensus_queue.cc:1035] T 60bae1191bee48dc846dab52864f700a P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:57.066551 29872 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:57.067981 30406 consensus_queue.cc:237] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:57.069528 32554 raft_consensus.cc:1261] T 28500f605e0644b69207077f57023ca6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:57.069873 32416 raft_consensus.cc:1261] T 28500f605e0644b69207077f57023ca6 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:57.069908 8439 consensus_queue.cc:1035] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:57.070288 8439 consensus_queue.cc:1035] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:09:57.070772 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Couldn't send request to peer d35c16dcb506426993811bb2519d7db3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28500f605e0644b69207077f57023ca6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:57.072932 8439 raft_consensus.cc:2932] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:57.073161 32533 tablet_service.cc:1467] Processing CreateTablet for tablet f5658528064c408786657396b5766dea (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d68d7841a4ab47978a724088f8e62a42]), partition=
I20240501 14:09:57.073179 30126 tablet_service.cc:1467] Processing CreateTablet for tablet f5658528064c408786657396b5766dea (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d68d7841a4ab47978a724088f8e62a42]), partition=
I20240501 14:09:57.073376 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5658528064c408786657396b5766dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.073397 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5658528064c408786657396b5766dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.074759 32554 raft_consensus.cc:2932] T 28500f605e0644b69207077f57023ca6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:57.075801 329 tablet_service.cc:1467] Processing CreateTablet for tablet f5658528064c408786657396b5766dea (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d68d7841a4ab47978a724088f8e62a42]), partition=
I20240501 14:09:57.075732 32416 raft_consensus.cc:2932] T 28500f605e0644b69207077f57023ca6 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:57.076069 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5658528064c408786657396b5766dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.079571 29873 catalog_manager.cc:5484] T 28500f605e0644b69207077f57023ca6 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:57.079700 8810 tablet_bootstrap.cc:492] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:57.081759 8811 tablet_bootstrap.cc:492] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:57.083765 8810 tablet_bootstrap.cc:654] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.082696 8811 tablet_bootstrap.cc:654] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.084861 8812 tablet_bootstrap.cc:492] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:57.085521 8811 tablet_bootstrap.cc:492] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:57.085971 8810 tablet_bootstrap.cc:492] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:57.086066 8810 ts_tablet_manager.cc:1367] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:57.086441 8812 tablet_bootstrap.cc:654] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.087597 8812 tablet_bootstrap.cc:492] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:57.087689 8812 ts_tablet_manager.cc:1367] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:57.087739 8811 ts_tablet_manager.cc:1367] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:57.088142 8812 raft_consensus.cc:348] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:57.088142 8811 raft_consensus.cc:348] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:57.088271 8811 raft_consensus.cc:374] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.088299 8812 raft_consensus.cc:374] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.088339 8812 raft_consensus.cc:729] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.088512 8812 consensus_queue.cc:260] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:57.088995 8810 raft_consensus.cc:348] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:57.089145 8811 raft_consensus.cc:729] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.089323 8811 consensus_queue.cc:260] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:57.089722 8811 ts_tablet_manager.cc:1398] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:57.089809 8812 ts_tablet_manager.cc:1398] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:57.089815 8810 raft_consensus.cc:374] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.089859 8810 raft_consensus.cc:729] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.089987 8810 consensus_queue.cc:260] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:57.090394 8810 ts_tablet_manager.cc:1398] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:57.098399 7999 consensus_queue.cc:1035] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:57.101895 7305 consensus_queue.cc:1035] T 838d383811344878812c688aa39df801 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:57.109424 8737 ts_tablet_manager.cc:902] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7: Initiating tablet copy from peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:57.109396 32683 consensus_queue.cc:237] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:57.109964 8737 tablet_copy_client.cc:308] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.135:41915
I20240501 14:09:57.110663 32705 tablet_copy_service.cc:140] P db935501e9104de5b15c8780681d6c38: Received BeginTabletCopySession request for tablet 8bb9687045d44efa888a69c7acde1105 from peer d1c217469ae4478ebd49e4ed9f7352d7 ({username='slave'} at 127.22.252.132:33995)
I20240501 14:09:57.110766 32705 tablet_copy_service.cc:161] P db935501e9104de5b15c8780681d6c38: Beginning new tablet copy session on tablet 8bb9687045d44efa888a69c7acde1105 from peer d1c217469ae4478ebd49e4ed9f7352d7 at {username='slave'} at 127.22.252.132:33995: session id = d1c217469ae4478ebd49e4ed9f7352d7-8bb9687045d44efa888a69c7acde1105
I20240501 14:09:57.111023 32554 raft_consensus.cc:1261] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:57.111626 32620 consensus_peers.cc:485] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 -> Peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): Couldn't send request to peer 93f2518fbadf4f9991cda93afacb363e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65bc0e6ad1fa491e8e5ae225f86561f9. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:57.111835 32705 tablet_copy_source_session.cc:205] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:57.111965 347 raft_consensus.cc:1261] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:57.112155 8511 consensus_queue.cc:1035] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:57.112494 8512 consensus_queue.cc:1035] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:57.112879 8737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb9687045d44efa888a69c7acde1105. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.114629 8511 raft_consensus.cc:2932] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:57.115135 8737 tablet_copy_client.cc:783] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:57.115319 8737 tablet_copy_client.cc:647] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:57.117980 347 raft_consensus.cc:2932] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:57.118927 29871 catalog_manager.cc:5484] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 reported cstate change: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:57.119540 32554 raft_consensus.cc:2932] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:57.120724 8737 tablet_copy_client.cc:523] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:57.122509 8737 tablet_bootstrap.cc:492] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:57.122934 8769 ts_tablet_manager.cc:902] T 33de572fa35c40109cb70ee3287829f2 P 93f2518fbadf4f9991cda93afacb363e: Initiating tablet copy from peer 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:57.122994 8842 ts_tablet_manager.cc:902] T b0ef35582c4f4d5c95199e00ccac92c9 P 1d0bb69611af4876a2b3b33c4766e13a: Initiating tablet copy from peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:57.123399 8769 tablet_copy_client.cc:308] T 33de572fa35c40109cb70ee3287829f2 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.136:45937
I20240501 14:09:57.123685 8842 tablet_copy_client.cc:308] T b0ef35582c4f4d5c95199e00ccac92c9 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.132:34357
I20240501 14:09:57.124155 369 tablet_copy_service.cc:140] P 9ecd907cf5634ac98f3f54f3dbec6849: Received BeginTabletCopySession request for tablet 33de572fa35c40109cb70ee3287829f2 from peer 93f2518fbadf4f9991cda93afacb363e ({username='slave'} at 127.22.252.129:40869)
I20240501 14:09:57.124277 369 tablet_copy_service.cc:161] P 9ecd907cf5634ac98f3f54f3dbec6849: Beginning new tablet copy session on tablet 33de572fa35c40109cb70ee3287829f2 from peer 93f2518fbadf4f9991cda93afacb363e at {username='slave'} at 127.22.252.129:40869: session id = 93f2518fbadf4f9991cda93afacb363e-33de572fa35c40109cb70ee3287829f2
I20240501 14:09:57.124861 30428 tablet_copy_service.cc:140] P d1c217469ae4478ebd49e4ed9f7352d7: Received BeginTabletCopySession request for tablet b0ef35582c4f4d5c95199e00ccac92c9 from peer 1d0bb69611af4876a2b3b33c4766e13a ({username='slave'} at 127.22.252.131:43493)
I20240501 14:09:57.124943 30428 tablet_copy_service.cc:161] P d1c217469ae4478ebd49e4ed9f7352d7: Beginning new tablet copy session on tablet b0ef35582c4f4d5c95199e00ccac92c9 from peer 1d0bb69611af4876a2b3b33c4766e13a at {username='slave'} at 127.22.252.131:43493: session id = 1d0bb69611af4876a2b3b33c4766e13a-b0ef35582c4f4d5c95199e00ccac92c9
I20240501 14:09:57.125015 8706 ts_tablet_manager.cc:902] T cfc701c19cd74d0db2f90ce580230177 P 9ecd907cf5634ac98f3f54f3dbec6849: Initiating tablet copy from peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:57.125247 369 tablet_copy_source_session.cc:205] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:57.125487 8706 tablet_copy_client.cc:308] T cfc701c19cd74d0db2f90ce580230177 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.132:34357
I20240501 14:09:57.125825 30428 tablet_copy_source_session.cc:205] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:57.125929 30427 tablet_copy_service.cc:140] P d1c217469ae4478ebd49e4ed9f7352d7: Received BeginTabletCopySession request for tablet cfc701c19cd74d0db2f90ce580230177 from peer 9ecd907cf5634ac98f3f54f3dbec6849 ({username='slave'} at 127.22.252.136:38643)
I20240501 14:09:57.126029 30427 tablet_copy_service.cc:161] P d1c217469ae4478ebd49e4ed9f7352d7: Beginning new tablet copy session on tablet cfc701c19cd74d0db2f90ce580230177 from peer 9ecd907cf5634ac98f3f54f3dbec6849 at {username='slave'} at 127.22.252.136:38643: session id = 9ecd907cf5634ac98f3f54f3dbec6849-cfc701c19cd74d0db2f90ce580230177
I20240501 14:09:57.126065 8769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33de572fa35c40109cb70ee3287829f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.126791 30427 tablet_copy_source_session.cc:205] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:57.127943 8842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ef35582c4f4d5c95199e00ccac92c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.128324 8769 tablet_copy_client.cc:783] T 33de572fa35c40109cb70ee3287829f2 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:57.128458 8769 tablet_copy_client.cc:647] T 33de572fa35c40109cb70ee3287829f2 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:57.129015 8841 ts_tablet_manager.cc:902] T c362bd01a59b4bcf85eeb59713e58b64 P 5f8a0cfdfdca4148a75686dc88059cf3: Initiating tablet copy from peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:57.129650 8769 tablet_copy_client.cc:523] T 33de572fa35c40109cb70ee3287829f2 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:57.129819 8841 tablet_copy_client.cc:308] T c362bd01a59b4bcf85eeb59713e58b64 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.132:34357
I20240501 14:09:57.130273 30427 tablet_copy_service.cc:140] P d1c217469ae4478ebd49e4ed9f7352d7: Received BeginTabletCopySession request for tablet c362bd01a59b4bcf85eeb59713e58b64 from peer 5f8a0cfdfdca4148a75686dc88059cf3 ({username='slave'} at 127.22.252.134:53955)
I20240501 14:09:57.130362 30427 tablet_copy_service.cc:161] P d1c217469ae4478ebd49e4ed9f7352d7: Beginning new tablet copy session on tablet c362bd01a59b4bcf85eeb59713e58b64 from peer 5f8a0cfdfdca4148a75686dc88059cf3 at {username='slave'} at 127.22.252.134:53955: session id = 5f8a0cfdfdca4148a75686dc88059cf3-c362bd01a59b4bcf85eeb59713e58b64
I20240501 14:09:57.131027 8769 tablet_bootstrap.cc:492] T 33de572fa35c40109cb70ee3287829f2 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:57.131265 30427 tablet_copy_source_session.cc:205] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:57.131631 8706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc701c19cd74d0db2f90ce580230177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.132408 8841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c362bd01a59b4bcf85eeb59713e58b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.134866 8841 tablet_copy_client.cc:783] T c362bd01a59b4bcf85eeb59713e58b64 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:57.135092 8841 tablet_copy_client.cc:647] T c362bd01a59b4bcf85eeb59713e58b64 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:57.135648 8737 tablet_bootstrap.cc:492] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:57.135823 8769 tablet_bootstrap.cc:492] T 33de572fa35c40109cb70ee3287829f2 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:57.136063 8769 tablet_bootstrap.cc:492] T 33de572fa35c40109cb70ee3287829f2 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap complete.
I20240501 14:09:57.136260 8769 ts_tablet_manager.cc:1367] T 33de572fa35c40109cb70ee3287829f2 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:09:57.136397 8841 tablet_copy_client.cc:523] T c362bd01a59b4bcf85eeb59713e58b64 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:57.136831 8769 raft_consensus.cc:348] T 33de572fa35c40109cb70ee3287829f2 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:57.136955 8769 raft_consensus.cc:729] T 33de572fa35c40109cb70ee3287829f2 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.136981 8737 tablet_bootstrap.cc:492] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap complete.
I20240501 14:09:57.137099 8769 consensus_queue.cc:260] T 33de572fa35c40109cb70ee3287829f2 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:57.137173 8737 ts_tablet_manager.cc:1367] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240501 14:09:57.137773 8841 tablet_bootstrap.cc:492] T c362bd01a59b4bcf85eeb59713e58b64 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:57.137771 8737 raft_consensus.cc:348] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:57.137876 8737 raft_consensus.cc:729] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.138023 8737 consensus_queue.cc:260] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:57.138613 8769 ts_tablet_manager.cc:1398] T 33de572fa35c40109cb70ee3287829f2 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:57.139441 369 tablet_copy_service.cc:342] P 9ecd907cf5634ac98f3f54f3dbec6849: Request end of tablet copy session 93f2518fbadf4f9991cda93afacb363e-33de572fa35c40109cb70ee3287829f2 received from {username='slave'} at 127.22.252.129:40869
I20240501 14:09:57.139508 369 tablet_copy_service.cc:433] P 9ecd907cf5634ac98f3f54f3dbec6849: ending tablet copy session 93f2518fbadf4f9991cda93afacb363e-33de572fa35c40109cb70ee3287829f2 on tablet 33de572fa35c40109cb70ee3287829f2 with peer 93f2518fbadf4f9991cda93afacb363e
I20240501 14:09:57.139492 8737 ts_tablet_manager.cc:1398] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:57.140451 32705 tablet_copy_service.cc:342] P db935501e9104de5b15c8780681d6c38: Request end of tablet copy session d1c217469ae4478ebd49e4ed9f7352d7-8bb9687045d44efa888a69c7acde1105 received from {username='slave'} at 127.22.252.132:33995
I20240501 14:09:57.140537 32705 tablet_copy_service.cc:433] P db935501e9104de5b15c8780681d6c38: ending tablet copy session d1c217469ae4478ebd49e4ed9f7352d7-8bb9687045d44efa888a69c7acde1105 on tablet 8bb9687045d44efa888a69c7acde1105 with peer d1c217469ae4478ebd49e4ed9f7352d7
I20240501 14:09:57.138392 8842 tablet_copy_client.cc:783] T b0ef35582c4f4d5c95199e00ccac92c9 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:57.144671 8842 tablet_copy_client.cc:647] T b0ef35582c4f4d5c95199e00ccac92c9 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:57.144793 8706 tablet_copy_client.cc:783] T cfc701c19cd74d0db2f90ce580230177 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:57.144994 8706 tablet_copy_client.cc:647] T cfc701c19cd74d0db2f90ce580230177 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:57.146044 8842 tablet_copy_client.cc:523] T b0ef35582c4f4d5c95199e00ccac92c9 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:57.147395 8842 tablet_bootstrap.cc:492] T b0ef35582c4f4d5c95199e00ccac92c9 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:57.149946 8706 tablet_copy_client.cc:523] T cfc701c19cd74d0db2f90ce580230177 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:57.151227 8706 tablet_bootstrap.cc:492] T cfc701c19cd74d0db2f90ce580230177 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:57.160742 8842 tablet_bootstrap.cc:492] T b0ef35582c4f4d5c95199e00ccac92c9 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:57.161016 8842 tablet_bootstrap.cc:492] T b0ef35582c4f4d5c95199e00ccac92c9 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap complete.
I20240501 14:09:57.161177 8842 ts_tablet_manager.cc:1367] T b0ef35582c4f4d5c95199e00ccac92c9 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.003s
I20240501 14:09:57.161617 8842 raft_consensus.cc:348] T b0ef35582c4f4d5c95199e00ccac92c9 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.161737 8842 raft_consensus.cc:729] T b0ef35582c4f4d5c95199e00ccac92c9 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.161857 8842 consensus_queue.cc:260] T b0ef35582c4f4d5c95199e00ccac92c9 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.162120 8842 ts_tablet_manager.cc:1398] T b0ef35582c4f4d5c95199e00ccac92c9 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:57.163491 30427 tablet_copy_service.cc:342] P d1c217469ae4478ebd49e4ed9f7352d7: Request end of tablet copy session 1d0bb69611af4876a2b3b33c4766e13a-b0ef35582c4f4d5c95199e00ccac92c9 received from {username='slave'} at 127.22.252.131:43493
I20240501 14:09:57.163570 30427 tablet_copy_service.cc:433] P d1c217469ae4478ebd49e4ed9f7352d7: ending tablet copy session 1d0bb69611af4876a2b3b33c4766e13a-b0ef35582c4f4d5c95199e00ccac92c9 on tablet b0ef35582c4f4d5c95199e00ccac92c9 with peer 1d0bb69611af4876a2b3b33c4766e13a
I20240501 14:09:57.167944 7305 consensus_queue.cc:1035] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:57.174942 8841 tablet_bootstrap.cc:492] T c362bd01a59b4bcf85eeb59713e58b64 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:57.175344 8841 tablet_bootstrap.cc:492] T c362bd01a59b4bcf85eeb59713e58b64 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap complete.
I20240501 14:09:57.175978 8841 ts_tablet_manager.cc:1367] T c362bd01a59b4bcf85eeb59713e58b64 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.038s user 0.028s sys 0.000s
I20240501 14:09:57.176901 8841 raft_consensus.cc:348] T c362bd01a59b4bcf85eeb59713e58b64 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:57.177048 8841 raft_consensus.cc:729] T c362bd01a59b4bcf85eeb59713e58b64 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.177192 8841 consensus_queue.cc:260] T c362bd01a59b4bcf85eeb59713e58b64 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:57.178689 7305 consensus_queue.cc:1035] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:57.181933 8841 ts_tablet_manager.cc:1398] T c362bd01a59b4bcf85eeb59713e58b64 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:57.186806 30427 tablet_copy_service.cc:342] P d1c217469ae4478ebd49e4ed9f7352d7: Request end of tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-c362bd01a59b4bcf85eeb59713e58b64 received from {username='slave'} at 127.22.252.134:53955
I20240501 14:09:57.186892 30427 tablet_copy_service.cc:433] P d1c217469ae4478ebd49e4ed9f7352d7: ending tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-c362bd01a59b4bcf85eeb59713e58b64 on tablet c362bd01a59b4bcf85eeb59713e58b64 with peer 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:09:57.191012 8881 ts_tablet_manager.cc:902] T 940c8f54339f4a32be2942682a41c685 P a91029b90bf9415a89a2f70736759ad3: Initiating tablet copy from peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:57.196597 8881 tablet_copy_client.cc:308] T 940c8f54339f4a32be2942682a41c685 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.132:34357
I20240501 14:09:57.196650 8706 tablet_bootstrap.cc:492] T cfc701c19cd74d0db2f90ce580230177 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:57.196916 8706 tablet_bootstrap.cc:492] T cfc701c19cd74d0db2f90ce580230177 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap complete.
I20240501 14:09:57.197103 8706 ts_tablet_manager.cc:1367] T cfc701c19cd74d0db2f90ce580230177 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.046s user 0.021s sys 0.003s
I20240501 14:09:57.197156 30427 tablet_copy_service.cc:140] P d1c217469ae4478ebd49e4ed9f7352d7: Received BeginTabletCopySession request for tablet 940c8f54339f4a32be2942682a41c685 from peer a91029b90bf9415a89a2f70736759ad3 ({username='slave'} at 127.22.252.133:43857)
I20240501 14:09:57.197235 30427 tablet_copy_service.cc:161] P d1c217469ae4478ebd49e4ed9f7352d7: Beginning new tablet copy session on tablet 940c8f54339f4a32be2942682a41c685 from peer a91029b90bf9415a89a2f70736759ad3 at {username='slave'} at 127.22.252.133:43857: session id = a91029b90bf9415a89a2f70736759ad3-940c8f54339f4a32be2942682a41c685
I20240501 14:09:57.197526 8706 raft_consensus.cc:348] T cfc701c19cd74d0db2f90ce580230177 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:57.197674 8706 raft_consensus.cc:729] T cfc701c19cd74d0db2f90ce580230177 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.197788 8706 consensus_queue.cc:260] T cfc701c19cd74d0db2f90ce580230177 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:57.198078 8706 ts_tablet_manager.cc:1398] T cfc701c19cd74d0db2f90ce580230177 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:57.198182 30427 tablet_copy_source_session.cc:205] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:57.199122 8881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940c8f54339f4a32be2942682a41c685. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.199242 30427 tablet_copy_service.cc:342] P d1c217469ae4478ebd49e4ed9f7352d7: Request end of tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-cfc701c19cd74d0db2f90ce580230177 received from {username='slave'} at 127.22.252.136:38643
I20240501 14:09:57.199303 30427 tablet_copy_service.cc:433] P d1c217469ae4478ebd49e4ed9f7352d7: ending tablet copy session 9ecd907cf5634ac98f3f54f3dbec6849-cfc701c19cd74d0db2f90ce580230177 on tablet cfc701c19cd74d0db2f90ce580230177 with peer 9ecd907cf5634ac98f3f54f3dbec6849
I20240501 14:09:57.203392 8881 tablet_copy_client.cc:783] T 940c8f54339f4a32be2942682a41c685 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:57.203564 8881 tablet_copy_client.cc:647] T 940c8f54339f4a32be2942682a41c685 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:57.207991 8881 tablet_copy_client.cc:523] T 940c8f54339f4a32be2942682a41c685 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:57.209277 8881 tablet_bootstrap.cc:492] T 940c8f54339f4a32be2942682a41c685 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:57.247267 8881 tablet_bootstrap.cc:492] T 940c8f54339f4a32be2942682a41c685 P a91029b90bf9415a89a2f70736759ad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:57.247629 8881 tablet_bootstrap.cc:492] T 940c8f54339f4a32be2942682a41c685 P a91029b90bf9415a89a2f70736759ad3: Bootstrap complete.
I20240501 14:09:57.247836 8881 ts_tablet_manager.cc:1367] T 940c8f54339f4a32be2942682a41c685 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.039s user 0.025s sys 0.002s
I20240501 14:09:57.250723 8881 raft_consensus.cc:348] T 940c8f54339f4a32be2942682a41c685 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:57.250860 8881 raft_consensus.cc:729] T 940c8f54339f4a32be2942682a41c685 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.251008 8881 consensus_queue.cc:260] T 940c8f54339f4a32be2942682a41c685 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:57.251394 8881 ts_tablet_manager.cc:1398] T 940c8f54339f4a32be2942682a41c685 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:09:57.251888 30427 tablet_copy_service.cc:342] P d1c217469ae4478ebd49e4ed9f7352d7: Request end of tablet copy session a91029b90bf9415a89a2f70736759ad3-940c8f54339f4a32be2942682a41c685 received from {username='slave'} at 127.22.252.133:43857
I20240501 14:09:57.251967 30427 tablet_copy_service.cc:433] P d1c217469ae4478ebd49e4ed9f7352d7: ending tablet copy session a91029b90bf9415a89a2f70736759ad3-940c8f54339f4a32be2942682a41c685 on tablet 940c8f54339f4a32be2942682a41c685 with peer a91029b90bf9415a89a2f70736759ad3
I20240501 14:09:57.267194 29872 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:57.274452 30126 tablet_service.cc:1467] Processing CreateTablet for tablet 825c128694ab47b1815bdb70b01b8a0a (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=54aa0217e9814060a0a77eccefebd53e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:57.274452 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 7534d5788f96488fbad5f6334f476224 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=54aa0217e9814060a0a77eccefebd53e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:57.274739 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 825c128694ab47b1815bdb70b01b8a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.274739 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7534d5788f96488fbad5f6334f476224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.275128 32664 tablet_service.cc:1467] Processing CreateTablet for tablet d1b4aad07d454290a175352f1379393d (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=54aa0217e9814060a0a77eccefebd53e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:57.275410 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1b4aad07d454290a175352f1379393d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.276531 30257 tablet_service.cc:1467] Processing CreateTablet for tablet 825c128694ab47b1815bdb70b01b8a0a (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=54aa0217e9814060a0a77eccefebd53e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:57.276844 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 825c128694ab47b1815bdb70b01b8a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.276973 8810 tablet_bootstrap.cc:492] T 825c128694ab47b1815bdb70b01b8a0a P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:57.277149 30255 tablet_service.cc:1467] Processing CreateTablet for tablet 7534d5788f96488fbad5f6334f476224 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=54aa0217e9814060a0a77eccefebd53e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:57.277346 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7534d5788f96488fbad5f6334f476224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.278139 8810 tablet_bootstrap.cc:654] T 825c128694ab47b1815bdb70b01b8a0a P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.278681 8913 tablet_bootstrap.cc:492] T d1b4aad07d454290a175352f1379393d P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:57.279551 8915 tablet_bootstrap.cc:492] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:57.279551 29994 tablet_service.cc:1467] Processing CreateTablet for tablet 825c128694ab47b1815bdb70b01b8a0a (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=54aa0217e9814060a0a77eccefebd53e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:57.279827 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 825c128694ab47b1815bdb70b01b8a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.280081 8737 ts_tablet_manager.cc:902] T 5db25d87e72347f4a4410bcde440f359 P d1c217469ae4478ebd49e4ed9f7352d7: Initiating tablet copy from peer 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:57.280305 8810 tablet_bootstrap.cc:492] T 825c128694ab47b1815bdb70b01b8a0a P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:57.280381 8810 ts_tablet_manager.cc:1367] T 825c128694ab47b1815bdb70b01b8a0a P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:57.280565 8737 tablet_copy_client.cc:308] T 5db25d87e72347f4a4410bcde440f359 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.136:45937
I20240501 14:09:57.280774 8810 raft_consensus.cc:348] T 825c128694ab47b1815bdb70b01b8a0a P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.280892 8810 raft_consensus.cc:374] T 825c128694ab47b1815bdb70b01b8a0a P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.280923 8810 raft_consensus.cc:729] T 825c128694ab47b1815bdb70b01b8a0a P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.281044 8810 consensus_queue.cc:260] T 825c128694ab47b1815bdb70b01b8a0a P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.281315 369 tablet_copy_service.cc:140] P 9ecd907cf5634ac98f3f54f3dbec6849: Received BeginTabletCopySession request for tablet 5db25d87e72347f4a4410bcde440f359 from peer d1c217469ae4478ebd49e4ed9f7352d7 ({username='slave'} at 127.22.252.132:39439)
I20240501 14:09:57.281392 8810 ts_tablet_manager.cc:1398] T 825c128694ab47b1815bdb70b01b8a0a P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:57.281412 369 tablet_copy_service.cc:161] P 9ecd907cf5634ac98f3f54f3dbec6849: Beginning new tablet copy session on tablet 5db25d87e72347f4a4410bcde440f359 from peer d1c217469ae4478ebd49e4ed9f7352d7 at {username='slave'} at 127.22.252.132:39439: session id = d1c217469ae4478ebd49e4ed9f7352d7-5db25d87e72347f4a4410bcde440f359
I20240501 14:09:57.282346 369 tablet_copy_source_session.cc:205] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:57.282420 29993 tablet_service.cc:1467] Processing CreateTablet for tablet d1b4aad07d454290a175352f1379393d (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=54aa0217e9814060a0a77eccefebd53e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:57.282663 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1b4aad07d454290a175352f1379393d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.282964 30126 tablet_service.cc:1467] Processing CreateTablet for tablet d1b4aad07d454290a175352f1379393d (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=54aa0217e9814060a0a77eccefebd53e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:57.283201 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1b4aad07d454290a175352f1379393d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.283489 8737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5db25d87e72347f4a4410bcde440f359. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.284991 32533 tablet_service.cc:1467] Processing CreateTablet for tablet 7534d5788f96488fbad5f6334f476224 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=54aa0217e9814060a0a77eccefebd53e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:57.285116 8881 ts_tablet_manager.cc:902] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3: Initiating tablet copy from peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:57.285167 8810 tablet_bootstrap.cc:492] T d1b4aad07d454290a175352f1379393d P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:57.285246 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7534d5788f96488fbad5f6334f476224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.285624 8737 tablet_copy_client.cc:783] T 5db25d87e72347f4a4410bcde440f359 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:57.285686 8881 tablet_copy_client.cc:308] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.130:36457
I20240501 14:09:57.285764 8737 tablet_copy_client.cc:647] T 5db25d87e72347f4a4410bcde440f359 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:57.286054 8810 tablet_bootstrap.cc:654] T d1b4aad07d454290a175352f1379393d P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.286089 30165 tablet_copy_service.cc:140] P d35c16dcb506426993811bb2519d7db3: Received BeginTabletCopySession request for tablet 5518112449d9448e98f5c779f1fa3248 from peer a91029b90bf9415a89a2f70736759ad3 ({username='slave'} at 127.22.252.133:37313)
I20240501 14:09:57.286155 30165 tablet_copy_service.cc:161] P d35c16dcb506426993811bb2519d7db3: Beginning new tablet copy session on tablet 5518112449d9448e98f5c779f1fa3248 from peer a91029b90bf9415a89a2f70736759ad3 at {username='slave'} at 127.22.252.133:37313: session id = a91029b90bf9415a89a2f70736759ad3-5518112449d9448e98f5c779f1fa3248
I20240501 14:09:57.286737 8915 tablet_bootstrap.cc:654] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.287026 30165 tablet_copy_source_session.cc:205] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:57.287652 8811 tablet_bootstrap.cc:492] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:57.288698 8811 tablet_bootstrap.cc:654] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.289261 8915 tablet_bootstrap.cc:492] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:57.289341 8915 ts_tablet_manager.cc:1367] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:57.289777 8915 raft_consensus.cc:348] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.289876 8915 raft_consensus.cc:374] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.289909 8915 raft_consensus.cc:729] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.290050 8915 consensus_queue.cc:260] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.290351 8915 ts_tablet_manager.cc:1398] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:57.290513 8915 tablet_bootstrap.cc:492] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:57.290724 8881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5518112449d9448e98f5c779f1fa3248. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.291316 8915 tablet_bootstrap.cc:654] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.295047 8881 tablet_copy_client.cc:783] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:57.295197 8881 tablet_copy_client.cc:647] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:57.297051 8913 tablet_bootstrap.cc:654] T d1b4aad07d454290a175352f1379393d P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.298223 8920 tablet_bootstrap.cc:492] T 825c128694ab47b1815bdb70b01b8a0a P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:57.299321 8920 tablet_bootstrap.cc:654] T 825c128694ab47b1815bdb70b01b8a0a P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.302455 8810 tablet_bootstrap.cc:492] T d1b4aad07d454290a175352f1379393d P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:57.302580 8810 ts_tablet_manager.cc:1367] T d1b4aad07d454290a175352f1379393d P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 14:09:57.303035 8810 raft_consensus.cc:348] T d1b4aad07d454290a175352f1379393d P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:57.303174 8810 raft_consensus.cc:374] T d1b4aad07d454290a175352f1379393d P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.303215 8810 raft_consensus.cc:729] T d1b4aad07d454290a175352f1379393d P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.303334 8810 consensus_queue.cc:260] T d1b4aad07d454290a175352f1379393d P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:57.303648 8810 ts_tablet_manager.cc:1398] T d1b4aad07d454290a175352f1379393d P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:57.304191 8881 tablet_copy_client.cc:523] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:57.305699 8881 tablet_bootstrap.cc:492] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:57.317571 8737 tablet_copy_client.cc:523] T 5db25d87e72347f4a4410bcde440f359 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:57.319126 8737 tablet_bootstrap.cc:492] T 5db25d87e72347f4a4410bcde440f359 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:57.318364 8920 tablet_bootstrap.cc:492] T 825c128694ab47b1815bdb70b01b8a0a P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:57.319886 8920 ts_tablet_manager.cc:1367] T 825c128694ab47b1815bdb70b01b8a0a P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240501 14:09:57.321218 8005 raft_consensus.cc:482] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:57.321314 8005 raft_consensus.cc:504] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.321707 8005 leader_election.cc:288] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:57.322206 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825c128694ab47b1815bdb70b01b8a0a" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:57.322242 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825c128694ab47b1815bdb70b01b8a0a" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:57.322341 30145 raft_consensus.cc:2454] T 825c128694ab47b1815bdb70b01b8a0a P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 0.
I20240501 14:09:57.322846 30213 leader_election.cc:302] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
W20240501 14:09:57.323354 30212 leader_election.cc:341] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:57.323902 8005 raft_consensus.cc:2781] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:57.323959 8005 raft_consensus.cc:482] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:57.324004 8005 raft_consensus.cc:3037] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.325069 8005 raft_consensus.cc:504] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.325196 8737 tablet_bootstrap.cc:492] T 5db25d87e72347f4a4410bcde440f359 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:57.325443 8737 tablet_bootstrap.cc:492] T 5db25d87e72347f4a4410bcde440f359 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap complete.
I20240501 14:09:57.325464 8005 leader_election.cc:288] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Requested vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:57.325584 8737 ts_tablet_manager.cc:1367] T 5db25d87e72347f4a4410bcde440f359 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240501 14:09:57.326289 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825c128694ab47b1815bdb70b01b8a0a" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:57.326424 30145 raft_consensus.cc:3037] T 825c128694ab47b1815bdb70b01b8a0a P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.326630 8737 raft_consensus.cc:348] T 5db25d87e72347f4a4410bcde440f359 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:57.326738 8737 raft_consensus.cc:729] T 5db25d87e72347f4a4410bcde440f359 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.327004 8737 consensus_queue.cc:260] T 5db25d87e72347f4a4410bcde440f359 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:57.327378 8737 ts_tablet_manager.cc:1398] T 5db25d87e72347f4a4410bcde440f359 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:57.327546 30145 raft_consensus.cc:2454] T 825c128694ab47b1815bdb70b01b8a0a P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:09:57.327807 30213 leader_election.cc:302] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:57.327977 8005 raft_consensus.cc:2781] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:57.328058 8005 raft_consensus.cc:686] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:09:57.328171 8005 consensus_queue.cc:237] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.328416 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825c128694ab47b1815bdb70b01b8a0a" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:57.328837 8920 raft_consensus.cc:348] T 825c128694ab47b1815bdb70b01b8a0a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.328954 8920 raft_consensus.cc:374] T 825c128694ab47b1815bdb70b01b8a0a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240501 14:09:57.328940 30212 leader_election.cc:341] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:57.328989 8920 raft_consensus.cc:729] T 825c128694ab47b1815bdb70b01b8a0a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.329152 8920 consensus_queue.cc:260] T 825c128694ab47b1815bdb70b01b8a0a P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.329519 8920 ts_tablet_manager.cc:1398] T 825c128694ab47b1815bdb70b01b8a0a P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240501 14:09:57.329677 8920 tablet_bootstrap.cc:492] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:57.330375 369 tablet_copy_service.cc:342] P 9ecd907cf5634ac98f3f54f3dbec6849: Request end of tablet copy session d1c217469ae4478ebd49e4ed9f7352d7-5db25d87e72347f4a4410bcde440f359 received from {username='slave'} at 127.22.252.132:39439
I20240501 14:09:57.330464 369 tablet_copy_service.cc:433] P 9ecd907cf5634ac98f3f54f3dbec6849: ending tablet copy session d1c217469ae4478ebd49e4ed9f7352d7-5db25d87e72347f4a4410bcde440f359 on tablet 5db25d87e72347f4a4410bcde440f359 with peer d1c217469ae4478ebd49e4ed9f7352d7
I20240501 14:09:57.331737 8920 tablet_bootstrap.cc:654] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.332320 8915 tablet_bootstrap.cc:492] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:57.332415 8915 ts_tablet_manager.cc:1367] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.000s
I20240501 14:09:57.332470 8913 tablet_bootstrap.cc:492] T d1b4aad07d454290a175352f1379393d P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:57.332437 29866 catalog_manager.cc:5484] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:57.332581 8913 ts_tablet_manager.cc:1367] T d1b4aad07d454290a175352f1379393d P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.000s
I20240501 14:09:57.332875 8915 raft_consensus.cc:348] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.332962 8915 raft_consensus.cc:374] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.332993 8915 raft_consensus.cc:729] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.333009 8913 raft_consensus.cc:348] T d1b4aad07d454290a175352f1379393d P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:57.333104 8913 raft_consensus.cc:374] T d1b4aad07d454290a175352f1379393d P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.333139 8913 raft_consensus.cc:729] T d1b4aad07d454290a175352f1379393d P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.333120 8915 consensus_queue.cc:260] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.333236 8913 consensus_queue.cc:260] T d1b4aad07d454290a175352f1379393d P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:57.333401 8915 ts_tablet_manager.cc:1398] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:57.333513 8913 ts_tablet_manager.cc:1398] T d1b4aad07d454290a175352f1379393d P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:57.333673 8913 tablet_bootstrap.cc:492] T 7534d5788f96488fbad5f6334f476224 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:57.334595 8913 tablet_bootstrap.cc:654] T 7534d5788f96488fbad5f6334f476224 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.336288 8913 tablet_bootstrap.cc:492] T 7534d5788f96488fbad5f6334f476224 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:57.336433 8913 ts_tablet_manager.cc:1367] T 7534d5788f96488fbad5f6334f476224 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:57.336966 8913 raft_consensus.cc:348] T 7534d5788f96488fbad5f6334f476224 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.337913 8913 raft_consensus.cc:374] T 7534d5788f96488fbad5f6334f476224 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.338030 8913 raft_consensus.cc:729] T 7534d5788f96488fbad5f6334f476224 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.338182 8913 consensus_queue.cc:260] T 7534d5788f96488fbad5f6334f476224 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.338477 8913 ts_tablet_manager.cc:1398] T 7534d5788f96488fbad5f6334f476224 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:57.339462 8811 tablet_bootstrap.cc:492] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:57.339561 8811 ts_tablet_manager.cc:1367] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.000s
I20240501 14:09:57.340041 8811 raft_consensus.cc:348] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.340157 8811 raft_consensus.cc:374] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.340195 8811 raft_consensus.cc:729] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.340346 8811 consensus_queue.cc:260] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.340657 8811 ts_tablet_manager.cc:1398] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:57.343129 8920 tablet_bootstrap.cc:492] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:57.343271 8920 ts_tablet_manager.cc:1367] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:09:57.343847 8920 raft_consensus.cc:348] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:57.343998 8920 raft_consensus.cc:374] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.344065 8920 raft_consensus.cc:729] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.344224 8920 consensus_queue.cc:260] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:57.345372 8920 ts_tablet_manager.cc:1398] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:57.355919 8769 ts_tablet_manager.cc:902] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e: Initiating tablet copy from peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:57.356418 8769 tablet_copy_client.cc:308] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.135:41915
I20240501 14:09:57.356833 32705 tablet_copy_service.cc:140] P db935501e9104de5b15c8780681d6c38: Received BeginTabletCopySession request for tablet 0cccc8eec30141c1946b57d3be27000f from peer 93f2518fbadf4f9991cda93afacb363e ({username='slave'} at 127.22.252.129:49273)
I20240501 14:09:57.356917 32705 tablet_copy_service.cc:161] P db935501e9104de5b15c8780681d6c38: Beginning new tablet copy session on tablet 0cccc8eec30141c1946b57d3be27000f from peer 93f2518fbadf4f9991cda93afacb363e at {username='slave'} at 127.22.252.129:49273: session id = 93f2518fbadf4f9991cda93afacb363e-0cccc8eec30141c1946b57d3be27000f
I20240501 14:09:57.357707 32705 tablet_copy_source_session.cc:205] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:57.358508 8769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cccc8eec30141c1946b57d3be27000f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.360061 8881 tablet_bootstrap.cc:492] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:57.360333 8881 tablet_bootstrap.cc:492] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3: Bootstrap complete.
I20240501 14:09:57.360374 8769 tablet_copy_client.cc:783] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:57.360503 8769 tablet_copy_client.cc:647] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:57.360523 8881 ts_tablet_manager.cc:1367] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.055s user 0.021s sys 0.003s
I20240501 14:09:57.360953 8881 raft_consensus.cc:348] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:57.361063 8881 raft_consensus.cc:729] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.361197 8881 consensus_queue.cc:260] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:57.361457 8769 tablet_copy_client.cc:523] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:57.361490 8881 ts_tablet_manager.cc:1398] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:57.361965 30165 tablet_copy_service.cc:342] P d35c16dcb506426993811bb2519d7db3: Request end of tablet copy session a91029b90bf9415a89a2f70736759ad3-5518112449d9448e98f5c779f1fa3248 received from {username='slave'} at 127.22.252.133:37313
I20240501 14:09:57.362038 30165 tablet_copy_service.cc:433] P d35c16dcb506426993811bb2519d7db3: ending tablet copy session a91029b90bf9415a89a2f70736759ad3-5518112449d9448e98f5c779f1fa3248 on tablet 5518112449d9448e98f5c779f1fa3248 with peer a91029b90bf9415a89a2f70736759ad3
I20240501 14:09:57.362607 8769 tablet_bootstrap.cc:492] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:57.365947 8769 tablet_bootstrap.cc:492] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:57.366218 8769 tablet_bootstrap.cc:492] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e: Bootstrap complete.
I20240501 14:09:57.366369 8769 ts_tablet_manager.cc:1367] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20240501 14:09:57.366668 8769 raft_consensus.cc:348] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:57.366746 8769 raft_consensus.cc:729] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.366840 8769 consensus_queue.cc:260] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:57.367110 8769 ts_tablet_manager.cc:1398] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:57.368016 32705 tablet_copy_service.cc:342] P db935501e9104de5b15c8780681d6c38: Request end of tablet copy session 93f2518fbadf4f9991cda93afacb363e-0cccc8eec30141c1946b57d3be27000f received from {username='slave'} at 127.22.252.129:49273
I20240501 14:09:57.368085 32705 tablet_copy_service.cc:433] P db935501e9104de5b15c8780681d6c38: ending tablet copy session 93f2518fbadf4f9991cda93afacb363e-0cccc8eec30141c1946b57d3be27000f on tablet 0cccc8eec30141c1946b57d3be27000f with peer 93f2518fbadf4f9991cda93afacb363e
I20240501 14:09:57.387334 8769 ts_tablet_manager.cc:902] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e: Initiating tablet copy from peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:57.387822 8769 tablet_copy_client.cc:308] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.135:41915
I20240501 14:09:57.388080 8842 ts_tablet_manager.cc:902] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a: Initiating tablet copy from peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:57.388586 32705 tablet_copy_service.cc:140] P db935501e9104de5b15c8780681d6c38: Received BeginTabletCopySession request for tablet 6208e4b3d25941848b2ca533edd5a7b9 from peer 93f2518fbadf4f9991cda93afacb363e ({username='slave'} at 127.22.252.129:49273)
I20240501 14:09:57.388623 8842 tablet_copy_client.cc:308] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.135:41915
I20240501 14:09:57.388689 32705 tablet_copy_service.cc:161] P db935501e9104de5b15c8780681d6c38: Beginning new tablet copy session on tablet 6208e4b3d25941848b2ca533edd5a7b9 from peer 93f2518fbadf4f9991cda93afacb363e at {username='slave'} at 127.22.252.129:49273: session id = 93f2518fbadf4f9991cda93afacb363e-6208e4b3d25941848b2ca533edd5a7b9
I20240501 14:09:57.389029 32704 tablet_copy_service.cc:140] P db935501e9104de5b15c8780681d6c38: Received BeginTabletCopySession request for tablet 7e337eb723974834b07aac5eacf73232 from peer 1d0bb69611af4876a2b3b33c4766e13a ({username='slave'} at 127.22.252.131:49203)
I20240501 14:09:57.389107 32704 tablet_copy_service.cc:161] P db935501e9104de5b15c8780681d6c38: Beginning new tablet copy session on tablet 7e337eb723974834b07aac5eacf73232 from peer 1d0bb69611af4876a2b3b33c4766e13a at {username='slave'} at 127.22.252.131:49203: session id = 1d0bb69611af4876a2b3b33c4766e13a-7e337eb723974834b07aac5eacf73232
I20240501 14:09:57.389137 347 raft_consensus.cc:1203] T dd2fd47ec6684cb28d2f4678d2e90e75 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 14:09:57.389668 32705 tablet_copy_source_session.cc:205] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:57.389956 32704 tablet_copy_source_session.cc:205] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:57.390740 8769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6208e4b3d25941848b2ca533edd5a7b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.390830 8842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e337eb723974834b07aac5eacf73232. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.392578 8769 tablet_copy_client.cc:783] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:57.392730 8769 tablet_copy_client.cc:647] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:57.392946 8842 tablet_copy_client.cc:783] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:57.393103 8842 tablet_copy_client.cc:647] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:57.393960 8769 tablet_copy_client.cc:523] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:57.394049 8842 tablet_copy_client.cc:523] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:57.395023 8769 tablet_bootstrap.cc:492] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:57.395072 8842 tablet_bootstrap.cc:492] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:57.398476 8769 tablet_bootstrap.cc:492] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:57.398628 8842 tablet_bootstrap.cc:492] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:57.398720 8769 tablet_bootstrap.cc:492] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap complete.
I20240501 14:09:57.398885 8769 ts_tablet_manager.cc:1367] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20240501 14:09:57.399001 8842 tablet_bootstrap.cc:492] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap complete.
I20240501 14:09:57.399150 8842 ts_tablet_manager.cc:1367] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:09:57.399384 8769 raft_consensus.cc:348] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:57.399502 8769 raft_consensus.cc:729] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.399529 8842 raft_consensus.cc:348] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.399642 8842 raft_consensus.cc:729] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.399652 8769 consensus_queue.cc:260] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:57.400071 8769 ts_tablet_manager.cc:1398] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240501 14:09:57.399775 8842 consensus_queue.cc:260] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.400683 32705 tablet_copy_service.cc:342] P db935501e9104de5b15c8780681d6c38: Request end of tablet copy session 93f2518fbadf4f9991cda93afacb363e-6208e4b3d25941848b2ca533edd5a7b9 received from {username='slave'} at 127.22.252.129:49273
I20240501 14:09:57.400763 32705 tablet_copy_service.cc:433] P db935501e9104de5b15c8780681d6c38: ending tablet copy session 93f2518fbadf4f9991cda93afacb363e-6208e4b3d25941848b2ca533edd5a7b9 on tablet 6208e4b3d25941848b2ca533edd5a7b9 with peer 93f2518fbadf4f9991cda93afacb363e
I20240501 14:09:57.401163 8842 ts_tablet_manager.cc:1398] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:57.401959 32705 tablet_copy_service.cc:342] P db935501e9104de5b15c8780681d6c38: Request end of tablet copy session 1d0bb69611af4876a2b3b33c4766e13a-7e337eb723974834b07aac5eacf73232 received from {username='slave'} at 127.22.252.131:49203
I20240501 14:09:57.402035 32705 tablet_copy_service.cc:433] P db935501e9104de5b15c8780681d6c38: ending tablet copy session 1d0bb69611af4876a2b3b33c4766e13a-7e337eb723974834b07aac5eacf73232 on tablet 7e337eb723974834b07aac5eacf73232 with peer 1d0bb69611af4876a2b3b33c4766e13a
I20240501 14:09:57.447252 8952 ts_tablet_manager.cc:902] T af39c304693645269a6af59190e95cad P db935501e9104de5b15c8780681d6c38: Initiating tablet copy from peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:57.447801 8952 tablet_copy_client.cc:308] T af39c304693645269a6af59190e95cad P db935501e9104de5b15c8780681d6c38: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.129:41387
I20240501 14:09:57.448297 30035 tablet_copy_service.cc:140] P 93f2518fbadf4f9991cda93afacb363e: Received BeginTabletCopySession request for tablet af39c304693645269a6af59190e95cad from peer db935501e9104de5b15c8780681d6c38 ({username='slave'} at 127.22.252.135:45653)
I20240501 14:09:57.448413 30035 tablet_copy_service.cc:161] P 93f2518fbadf4f9991cda93afacb363e: Beginning new tablet copy session on tablet af39c304693645269a6af59190e95cad from peer db935501e9104de5b15c8780681d6c38 at {username='slave'} at 127.22.252.135:45653: session id = db935501e9104de5b15c8780681d6c38-af39c304693645269a6af59190e95cad
I20240501 14:09:57.448854 8841 ts_tablet_manager.cc:902] T bf0712a5e11c46a291b90bfb491b8b25 P 5f8a0cfdfdca4148a75686dc88059cf3: Initiating tablet copy from peer 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:57.449319 8841 tablet_copy_client.cc:308] T bf0712a5e11c46a291b90bfb491b8b25 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.136:45937
I20240501 14:09:57.449462 30035 tablet_copy_source_session.cc:205] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:57.449884 369 tablet_copy_service.cc:140] P 9ecd907cf5634ac98f3f54f3dbec6849: Received BeginTabletCopySession request for tablet bf0712a5e11c46a291b90bfb491b8b25 from peer 5f8a0cfdfdca4148a75686dc88059cf3 ({username='slave'} at 127.22.252.134:51703)
I20240501 14:09:57.450001 369 tablet_copy_service.cc:161] P 9ecd907cf5634ac98f3f54f3dbec6849: Beginning new tablet copy session on tablet bf0712a5e11c46a291b90bfb491b8b25 from peer 5f8a0cfdfdca4148a75686dc88059cf3 at {username='slave'} at 127.22.252.134:51703: session id = 5f8a0cfdfdca4148a75686dc88059cf3-bf0712a5e11c46a291b90bfb491b8b25
I20240501 14:09:57.450281 8952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af39c304693645269a6af59190e95cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.450718 369 tablet_copy_source_session.cc:205] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:57.451470 8841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf0712a5e11c46a291b90bfb491b8b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.452368 8952 tablet_copy_client.cc:783] T af39c304693645269a6af59190e95cad P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:57.452528 8952 tablet_copy_client.cc:647] T af39c304693645269a6af59190e95cad P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:57.453711 8841 tablet_copy_client.cc:783] T bf0712a5e11c46a291b90bfb491b8b25 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:57.453862 8841 tablet_copy_client.cc:647] T bf0712a5e11c46a291b90bfb491b8b25 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:57.454420 8952 tablet_copy_client.cc:523] T af39c304693645269a6af59190e95cad P db935501e9104de5b15c8780681d6c38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:57.454946 8841 tablet_copy_client.cc:523] T bf0712a5e11c46a291b90bfb491b8b25 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:57.455803 8952 tablet_bootstrap.cc:492] T af39c304693645269a6af59190e95cad P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:57.456432 8841 tablet_bootstrap.cc:492] T bf0712a5e11c46a291b90bfb491b8b25 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:57.460139 8841 tablet_bootstrap.cc:492] T bf0712a5e11c46a291b90bfb491b8b25 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:57.460522 8841 tablet_bootstrap.cc:492] T bf0712a5e11c46a291b90bfb491b8b25 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap complete.
I20240501 14:09:57.460844 8841 ts_tablet_manager.cc:1367] T bf0712a5e11c46a291b90bfb491b8b25 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:09:57.461310 8841 raft_consensus.cc:348] T bf0712a5e11c46a291b90bfb491b8b25 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:57.461483 8841 raft_consensus.cc:729] T bf0712a5e11c46a291b90bfb491b8b25 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.461649 8841 consensus_queue.cc:260] T bf0712a5e11c46a291b90bfb491b8b25 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:57.462003 8841 ts_tablet_manager.cc:1398] T bf0712a5e11c46a291b90bfb491b8b25 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:57.462620 369 tablet_copy_service.cc:342] P 9ecd907cf5634ac98f3f54f3dbec6849: Request end of tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-bf0712a5e11c46a291b90bfb491b8b25 received from {username='slave'} at 127.22.252.134:51703
I20240501 14:09:57.462702 369 tablet_copy_service.cc:433] P 9ecd907cf5634ac98f3f54f3dbec6849: ending tablet copy session 5f8a0cfdfdca4148a75686dc88059cf3-bf0712a5e11c46a291b90bfb491b8b25 on tablet bf0712a5e11c46a291b90bfb491b8b25 with peer 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:09:57.466734 30408 raft_consensus.cc:1203] T 14267ff763f5409998f457e14e407ed0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:57.478106 8952 tablet_bootstrap.cc:492] T af39c304693645269a6af59190e95cad P db935501e9104de5b15c8780681d6c38: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:57.478400 8952 tablet_bootstrap.cc:492] T af39c304693645269a6af59190e95cad P db935501e9104de5b15c8780681d6c38: Bootstrap complete.
I20240501 14:09:57.478569 8952 ts_tablet_manager.cc:1367] T af39c304693645269a6af59190e95cad P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.023s user 0.021s sys 0.000s
I20240501 14:09:57.478993 8952 raft_consensus.cc:348] T af39c304693645269a6af59190e95cad P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:57.479091 8952 raft_consensus.cc:729] T af39c304693645269a6af59190e95cad P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.479223 8952 consensus_queue.cc:260] T af39c304693645269a6af59190e95cad P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:57.479528 8952 ts_tablet_manager.cc:1398] T af39c304693645269a6af59190e95cad P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:57.479976 30035 tablet_copy_service.cc:342] P 93f2518fbadf4f9991cda93afacb363e: Request end of tablet copy session db935501e9104de5b15c8780681d6c38-af39c304693645269a6af59190e95cad received from {username='slave'} at 127.22.252.135:45653
I20240501 14:09:57.480057 30035 tablet_copy_service.cc:433] P 93f2518fbadf4f9991cda93afacb363e: ending tablet copy session db935501e9104de5b15c8780681d6c38-af39c304693645269a6af59190e95cad on tablet af39c304693645269a6af59190e95cad with peer db935501e9104de5b15c8780681d6c38
I20240501 14:09:57.481325 30145 raft_consensus.cc:1203] T a4462d7eaaa54223998369121761acd6 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:57.483783 8102 raft_consensus.cc:482] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:57.483875 8102 raft_consensus.cc:504] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:57.484320 8102 leader_election.cc:288] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:57.484616 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5658528064c408786657396b5766dea" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:57.484650 347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5658528064c408786657396b5766dea" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:57.484740 30145 raft_consensus.cc:2454] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:57.484758 347 raft_consensus.cc:2454] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:57.484972 32489 leader_election.cc:302] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:57.485219 8102 raft_consensus.cc:2781] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:57.485273 8102 raft_consensus.cc:482] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:57.485329 8102 raft_consensus.cc:3037] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.486157 8102 raft_consensus.cc:504] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:57.486413 8102 leader_election.cc:288] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:57.486651 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5658528064c408786657396b5766dea" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:57.486753 30145 raft_consensus.cc:3037] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.486793 347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5658528064c408786657396b5766dea" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:57.486893 347 raft_consensus.cc:3037] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.487731 30145 raft_consensus.cc:2454] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:57.487923 347 raft_consensus.cc:2454] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:57.488032 32490 leader_election.cc:302] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:57.488359 8102 raft_consensus.cc:2781] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:57.488448 8102 raft_consensus.cc:686] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:09:57.488564 8102 consensus_queue.cc:237] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:57.490900 8964 ts_tablet_manager.cc:902] T 28500f605e0644b69207077f57023ca6 P d35c16dcb506426993811bb2519d7db3: Initiating tablet copy from peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:57.491346 8964 tablet_copy_client.cc:308] T 28500f605e0644b69207077f57023ca6 P d35c16dcb506426993811bb2519d7db3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.132:34357
I20240501 14:09:57.491708 30427 tablet_copy_service.cc:140] P d1c217469ae4478ebd49e4ed9f7352d7: Received BeginTabletCopySession request for tablet 28500f605e0644b69207077f57023ca6 from peer d35c16dcb506426993811bb2519d7db3 ({username='slave'} at 127.22.252.130:46573)
I20240501 14:09:57.491794 30427 tablet_copy_service.cc:161] P d1c217469ae4478ebd49e4ed9f7352d7: Beginning new tablet copy session on tablet 28500f605e0644b69207077f57023ca6 from peer d35c16dcb506426993811bb2519d7db3 at {username='slave'} at 127.22.252.130:46573: session id = d35c16dcb506426993811bb2519d7db3-28500f605e0644b69207077f57023ca6
I20240501 14:09:57.492142 29867 catalog_manager.cc:5484] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:57.492683 30427 tablet_copy_source_session.cc:205] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:57.494230 8964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28500f605e0644b69207077f57023ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.496378 8964 tablet_copy_client.cc:783] T 28500f605e0644b69207077f57023ca6 P d35c16dcb506426993811bb2519d7db3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:57.496522 8964 tablet_copy_client.cc:647] T 28500f605e0644b69207077f57023ca6 P d35c16dcb506426993811bb2519d7db3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:57.497519 8964 tablet_copy_client.cc:523] T 28500f605e0644b69207077f57023ca6 P d35c16dcb506426993811bb2519d7db3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:57.498566 8964 tablet_bootstrap.cc:492] T 28500f605e0644b69207077f57023ca6 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:57.499792 30408 raft_consensus.cc:1203] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:57.502213 8964 tablet_bootstrap.cc:492] T 28500f605e0644b69207077f57023ca6 P d35c16dcb506426993811bb2519d7db3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:57.502483 8964 tablet_bootstrap.cc:492] T 28500f605e0644b69207077f57023ca6 P d35c16dcb506426993811bb2519d7db3: Bootstrap complete.
I20240501 14:09:57.502660 8964 ts_tablet_manager.cc:1367] T 28500f605e0644b69207077f57023ca6 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:09:57.503118 8964 raft_consensus.cc:348] T 28500f605e0644b69207077f57023ca6 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:57.503268 8964 raft_consensus.cc:729] T 28500f605e0644b69207077f57023ca6 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.503428 8964 consensus_queue.cc:260] T 28500f605e0644b69207077f57023ca6 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:57.504442 8964 ts_tablet_manager.cc:1398] T 28500f605e0644b69207077f57023ca6 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:57.504874 30427 tablet_copy_service.cc:342] P d1c217469ae4478ebd49e4ed9f7352d7: Request end of tablet copy session d35c16dcb506426993811bb2519d7db3-28500f605e0644b69207077f57023ca6 received from {username='slave'} at 127.22.252.130:46573
I20240501 14:09:57.504952 30427 tablet_copy_service.cc:433] P d1c217469ae4478ebd49e4ed9f7352d7: ending tablet copy session d35c16dcb506426993811bb2519d7db3-28500f605e0644b69207077f57023ca6 on tablet 28500f605e0644b69207077f57023ca6 with peer d35c16dcb506426993811bb2519d7db3
I20240501 14:09:57.516022 347 raft_consensus.cc:1203] T cfc701c19cd74d0db2f90ce580230177 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 14:09:57.525765 29993 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:57.528784 30386 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:57.534492 32533 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:57.536525 8102 raft_consensus.cc:482] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:57.536624 8102 raft_consensus.cc:504] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.537035 329 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:57.537091 8102 leader_election.cc:288] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:57.537714 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7534d5788f96488fbad5f6334f476224" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:57.537846 32683 raft_consensus.cc:2454] T 7534d5788f96488fbad5f6334f476224 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:57.537832 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7534d5788f96488fbad5f6334f476224" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:57.537966 30264 raft_consensus.cc:2454] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:57.538369 32487 leader_election.cc:302] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:57.539952 8102 raft_consensus.cc:2781] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:57.540014 8102 raft_consensus.cc:482] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:57.540048 8102 raft_consensus.cc:3037] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.541121 8842 ts_tablet_manager.cc:902] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a: Initiating tablet copy from peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:57.541180 8102 raft_consensus.cc:504] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.541524 8102 leader_election.cc:288] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:57.541601 8842 tablet_copy_client.cc:308] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.135:41915
I20240501 14:09:57.542105 32705 tablet_copy_service.cc:140] P db935501e9104de5b15c8780681d6c38: Received BeginTabletCopySession request for tablet d8430bd8cb7549f0aa4ede9a3d1ffe6e from peer 1d0bb69611af4876a2b3b33c4766e13a ({username='slave'} at 127.22.252.131:49203)
I20240501 14:09:57.542204 32705 tablet_copy_service.cc:161] P db935501e9104de5b15c8780681d6c38: Beginning new tablet copy session on tablet d8430bd8cb7549f0aa4ede9a3d1ffe6e from peer 1d0bb69611af4876a2b3b33c4766e13a at {username='slave'} at 127.22.252.131:49203: session id = 1d0bb69611af4876a2b3b33c4766e13a-d8430bd8cb7549f0aa4ede9a3d1ffe6e
I20240501 14:09:57.542934 32705 tablet_copy_source_session.cc:205] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:57.543844 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7534d5788f96488fbad5f6334f476224" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:57.543947 32683 raft_consensus.cc:3037] T 7534d5788f96488fbad5f6334f476224 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.544145 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7534d5788f96488fbad5f6334f476224" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:57.544270 30277 raft_consensus.cc:3037] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.544952 32683 raft_consensus.cc:2454] T 7534d5788f96488fbad5f6334f476224 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:57.545336 30277 raft_consensus.cc:2454] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:57.545349 32487 leader_election.cc:302] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:57.545492 8102 raft_consensus.cc:2781] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:57.545560 8102 raft_consensus.cc:686] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:09:57.545684 8102 consensus_queue.cc:237] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.545982 8842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8430bd8cb7549f0aa4ede9a3d1ffe6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.548592 29867 catalog_manager.cc:5484] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:57.548089 8842 tablet_copy_client.cc:783] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:57.553236 8842 tablet_copy_client.cc:647] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:57.554407 32400 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:09:57.555506 30126 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:09:57.555488 30014 raft_consensus.cc:1203] T c7df6b8d76714eebbeacc85f403d3405 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:57.566042 8669 raft_consensus.cc:482] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:57.566138 8669 raft_consensus.cc:504] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:57.566488 8669 leader_election.cc:288] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:57.566937 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b4aad07d454290a175352f1379393d" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:57.566982 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b4aad07d454290a175352f1379393d" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:57.567066 32683 raft_consensus.cc:2454] T d1b4aad07d454290a175352f1379393d P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:57.567091 30145 raft_consensus.cc:2454] T d1b4aad07d454290a175352f1379393d P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:09:57.567373 29948 leader_election.cc:302] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:57.567634 8669 raft_consensus.cc:2781] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:57.567695 8669 raft_consensus.cc:482] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:57.567745 8669 raft_consensus.cc:3037] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.568699 8669 raft_consensus.cc:504] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:57.568997 8842 tablet_copy_client.cc:523] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:57.569049 8669 leader_election.cc:288] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:57.569684 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b4aad07d454290a175352f1379393d" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:57.569792 30145 raft_consensus.cc:3037] T d1b4aad07d454290a175352f1379393d P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.570394 8842 tablet_bootstrap.cc:492] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:57.570577 30145 raft_consensus.cc:2454] T d1b4aad07d454290a175352f1379393d P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:57.570914 29951 leader_election.cc:302] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:57.571072 7378 raft_consensus.cc:482] T 825c128694ab47b1815bdb70b01b8a0a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:57.571143 7378 raft_consensus.cc:504] T 825c128694ab47b1815bdb70b01b8a0a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.571257 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b4aad07d454290a175352f1379393d" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:57.571348 32683 raft_consensus.cc:3037] T d1b4aad07d454290a175352f1379393d P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.571501 7378 leader_election.cc:288] T 825c128694ab47b1815bdb70b01b8a0a P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:57.571579 7378 raft_consensus.cc:2781] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:57.571655 7378 raft_consensus.cc:686] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:09:57.571766 7378 consensus_queue.cc:237] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:57.572351 32683 raft_consensus.cc:2454] T d1b4aad07d454290a175352f1379393d P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:09:57.573169 32665 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:57.574855 8842 tablet_bootstrap.cc:492] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:57.575444 8842 tablet_bootstrap.cc:492] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap complete.
I20240501 14:09:57.575575 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825c128694ab47b1815bdb70b01b8a0a" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:57.575659 8842 ts_tablet_manager.cc:1367] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240501 14:09:57.576058 8842 raft_consensus.cc:348] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.576159 8842 raft_consensus.cc:729] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.576380 8842 consensus_queue.cc:260] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:57.576740 8842 ts_tablet_manager.cc:1398] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:57.577173 29869 catalog_manager.cc:5484] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:57.580801 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:57.581967 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825c128694ab47b1815bdb70b01b8a0a" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:57.582105 30145 raft_consensus.cc:2379] T 825c128694ab47b1815bdb70b01b8a0a P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93f2518fbadf4f9991cda93afacb363e in current term 1: Already voted for candidate 1d0bb69611af4876a2b3b33c4766e13a in this term.
I20240501 14:09:57.582422 29951 leader_election.cc:302] T 825c128694ab47b1815bdb70b01b8a0a P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3
I20240501 14:09:57.582674 8669 raft_consensus.cc:3037] T 825c128694ab47b1815bdb70b01b8a0a P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.583576 8669 raft_consensus.cc:2726] T 825c128694ab47b1815bdb70b01b8a0a P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:09:57.583720 32705 tablet_copy_service.cc:342] P db935501e9104de5b15c8780681d6c38: Request end of tablet copy session 1d0bb69611af4876a2b3b33c4766e13a-d8430bd8cb7549f0aa4ede9a3d1ffe6e received from {username='slave'} at 127.22.252.131:49203
I20240501 14:09:57.583812 32705 tablet_copy_service.cc:433] P db935501e9104de5b15c8780681d6c38: ending tablet copy session 1d0bb69611af4876a2b3b33c4766e13a-d8430bd8cb7549f0aa4ede9a3d1ffe6e on tablet d8430bd8cb7549f0aa4ede9a3d1ffe6e with peer 1d0bb69611af4876a2b3b33c4766e13a
I20240501 14:09:57.608851 8769 ts_tablet_manager.cc:902] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e: Initiating tablet copy from peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:09:57.609381 8769 tablet_copy_client.cc:308] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.135:41915
I20240501 14:09:57.609859 32705 tablet_copy_service.cc:140] P db935501e9104de5b15c8780681d6c38: Received BeginTabletCopySession request for tablet 65bc0e6ad1fa491e8e5ae225f86561f9 from peer 93f2518fbadf4f9991cda93afacb363e ({username='slave'} at 127.22.252.129:49273)
I20240501 14:09:57.609957 32705 tablet_copy_service.cc:161] P db935501e9104de5b15c8780681d6c38: Beginning new tablet copy session on tablet 65bc0e6ad1fa491e8e5ae225f86561f9 from peer 93f2518fbadf4f9991cda93afacb363e at {username='slave'} at 127.22.252.129:49273: session id = 93f2518fbadf4f9991cda93afacb363e-65bc0e6ad1fa491e8e5ae225f86561f9
I20240501 14:09:57.610927 32705 tablet_copy_source_session.cc:205] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:57.612787 8769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65bc0e6ad1fa491e8e5ae225f86561f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.617046 8769 tablet_copy_client.cc:783] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:57.617265 8769 tablet_copy_client.cc:647] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:57.643702 8769 tablet_copy_client.cc:523] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:57.645527 8769 tablet_bootstrap.cc:492] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:57.649194 8769 tablet_bootstrap.cc:492] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:57.649513 8769 tablet_bootstrap.cc:492] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap complete.
I20240501 14:09:57.649717 8769 ts_tablet_manager.cc:1367] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:09:57.650570 8769 raft_consensus.cc:348] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:57.651335 8769 raft_consensus.cc:729] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.651553 8769 consensus_queue.cc:260] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:57.652177 8769 ts_tablet_manager.cc:1398] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:57.653038 32705 tablet_copy_service.cc:342] P db935501e9104de5b15c8780681d6c38: Request end of tablet copy session 93f2518fbadf4f9991cda93afacb363e-65bc0e6ad1fa491e8e5ae225f86561f9 received from {username='slave'} at 127.22.252.129:49273
I20240501 14:09:57.653120 32705 tablet_copy_service.cc:433] P db935501e9104de5b15c8780681d6c38: ending tablet copy session 93f2518fbadf4f9991cda93afacb363e-65bc0e6ad1fa491e8e5ae225f86561f9 on tablet 65bc0e6ad1fa491e8e5ae225f86561f9 with peer 93f2518fbadf4f9991cda93afacb363e
I20240501 14:09:57.661792 29867 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:57.665601 349 raft_consensus.cc:1261] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:57.665975 8102 consensus_queue.cc:1035] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:57.667248 30145 raft_consensus.cc:1261] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:57.667862 7853 consensus_queue.cc:1035] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:57.669530 29866 catalog_manager.cc:5576] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937) does not have the latest schema for tablet f5658528064c408786657396b5766dea (table rebalancer_dynamic_table [id=d68d7841a4ab47978a724088f8e62a42]). Expected version 1 got 0
I20240501 14:09:57.671988 29867 catalog_manager.cc:5576] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457) does not have the latest schema for tablet f5658528064c408786657396b5766dea (table rebalancer_dynamic_table [id=d68d7841a4ab47978a724088f8e62a42]). Expected version 1 got 0
I20240501 14:09:57.673333 9008 tablet.cc:1722] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:57.673736 9008 tablet.cc:1620] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:57.673708 9011 tablet.cc:1722] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:57.674047 9011 tablet.cc:1620] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:57.675063 29866 catalog_manager.cc:5576] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937) does not have the latest schema for tablet f5658528064c408786657396b5766dea (table rebalancer_dynamic_table [id=d68d7841a4ab47978a724088f8e62a42]). Expected version 1 got 0
I20240501 14:09:57.675494 29867 catalog_manager.cc:5576] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457) does not have the latest schema for tablet f5658528064c408786657396b5766dea (table rebalancer_dynamic_table [id=d68d7841a4ab47978a724088f8e62a42]). Expected version 1 got 0
I20240501 14:09:57.677644 9016 tablet.cc:1722] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:57.679349 9016 tablet.cc:1620] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3: MemRowSet was empty: no flush needed.
I20240501 14:09:57.679865 29867 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=d68d7841a4ab47978a724088f8e62a42] alter complete (version 1)
I20240501 14:09:57.691542 32554 raft_consensus.cc:1203] T c362bd01a59b4bcf85eeb59713e58b64 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.6->[2.7-2.7] Dedup: 2.7->[]
I20240501 14:09:57.701941 30277 raft_consensus.cc:1203] T b0ef35582c4f4d5c95199e00ccac92c9 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:57.717125 30014 raft_consensus.cc:1203] T 33de572fa35c40109cb70ee3287829f2 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:57.772768 30014 raft_consensus.cc:1203] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:57.781884 32416 raft_consensus.cc:1203] T 940c8f54339f4a32be2942682a41c685 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 14:09:57.784516 30277 raft_consensus.cc:1203] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:57.785959 29868 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:57.801885 9008 tablet.cc:1722] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:57.802243 9008 tablet.cc:1620] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:57.803109 30406 raft_consensus.cc:1203] T 5db25d87e72347f4a4410bcde440f359 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:57.805632 9011 tablet.cc:1722] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:57.805974 9011 tablet.cc:1620] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:57.810220 9016 tablet.cc:1722] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:57.810595 9016 tablet.cc:1620] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3: MemRowSet was empty: no flush needed.
I20240501 14:09:57.816035 32665 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:57.825017 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=d68d7841a4ab47978a724088f8e62a42] alter complete (version 2)
I20240501 14:09:57.827445 32533 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:57.827808 30386 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:57.835691 32400 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:09:57.835830 329 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:57.836227 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:57.839488 30126 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:09:57.844337 32552 raft_consensus.cc:1203] T bf0712a5e11c46a291b90bfb491b8b25 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:57.845078 32416 raft_consensus.cc:1203] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 14:09:57.866387 30014 raft_consensus.cc:1203] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:57.883718 32683 raft_consensus.cc:1203] T af39c304693645269a6af59190e95cad P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 14:09:57.885370 29993 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:57.895905 8005 consensus_queue.cc:1035] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:57.912119 8437 consensus_queue.cc:1035] T 825c128694ab47b1815bdb70b01b8a0a P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:57.930995 29868 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:57.934705 30277 raft_consensus.cc:1203] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:57.938541 9008 tablet.cc:1722] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:57.939893 9011 tablet.cc:1722] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:57.940184 9016 tablet.cc:1722] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:57.946561 29868 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=d68d7841a4ab47978a724088f8e62a42] alter complete (version 3)
I20240501 14:09:57.954890 30145 raft_consensus.cc:1203] T 28500f605e0644b69207077f57023ca6 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:57.978441 8669 consensus_queue.cc:1035] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:57.983562 7378 consensus_queue.cc:1035] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:58.051707 29868 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:58.052754 29868 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:58.055697 29868 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet f5658528064c408786657396b5766dea
I20240501 14:09:58.056591 329 tablet_service.cc:1514] Processing DeleteTablet for tablet f5658528064c408786657396b5766dea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:58 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:58.059201 30126 tablet_service.cc:1514] Processing DeleteTablet for tablet f5658528064c408786657396b5766dea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:58 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:58.059695 32533 tablet_service.cc:1514] Processing DeleteTablet for tablet f5658528064c408786657396b5766dea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:58 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:58.059988 9111 tablet_replica.cc:330] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:58.060196 9111 raft_consensus.cc:2229] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:58.060477 9111 raft_consensus.cc:2258] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:58.061199 9109 tablet_replica.cc:330] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:58.061446 9111 ts_tablet_manager.cc:1875] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:58.061770 9110 tablet_replica.cc:330] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:58.061911 9110 raft_consensus.cc:2229] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:58.062016 9110 raft_consensus.cc:2258] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:58.062228 9109 raft_consensus.cc:2229] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:58.062361 9109 raft_consensus.cc:2258] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:58.062955 9109 ts_tablet_manager.cc:1875] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:58.063589 9110 ts_tablet_manager.cc:1875] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:58.067523 9109 ts_tablet_manager.cc:1888] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:58.068900 9110 ts_tablet_manager.cc:1888] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:58.068980 9110 log.cc:1196] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/f5658528064c408786657396b5766dea
I20240501 14:09:58.069087 9109 log.cc:1196] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/f5658528064c408786657396b5766dea
I20240501 14:09:58.069406 9109 ts_tablet_manager.cc:1909] T f5658528064c408786657396b5766dea P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:09:58.070178 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet f5658528064c408786657396b5766dea (table rebalancer_dynamic_table_new_name [id=d68d7841a4ab47978a724088f8e62a42]) successfully deleted
I20240501 14:09:58.070319 9110 ts_tablet_manager.cc:1909] T f5658528064c408786657396b5766dea P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:58.071131 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet f5658528064c408786657396b5766dea (table rebalancer_dynamic_table_new_name [id=d68d7841a4ab47978a724088f8e62a42]) successfully deleted
I20240501 14:09:58.075042 9111 ts_tablet_manager.cc:1888] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:58.075129 9111 log.cc:1196] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/f5658528064c408786657396b5766dea
I20240501 14:09:58.075448 9111 ts_tablet_manager.cc:1909] T f5658528064c408786657396b5766dea P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:58.076155 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet f5658528064c408786657396b5766dea (table rebalancer_dynamic_table_new_name [id=d68d7841a4ab47978a724088f8e62a42]) successfully deleted
I20240501 14:09:58.100526 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:58.101233 30386 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:58.102458 329 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:58.104162 30126 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:09:58.105767 32400 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:09:58.112722 29872 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:58.121186 32533 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:58.128778 32400 tablet_service.cc:1467] Processing CreateTablet for tablet efeb0f330ef04192999e217612c4461f (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=1b4f1f3f44114f5b8700b7074f2f63d1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.129122 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efeb0f330ef04192999e217612c4461f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.129160 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 27c905e41bfb4d99af6491331b588c06 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=1b4f1f3f44114f5b8700b7074f2f63d1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.129477 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c905e41bfb4d99af6491331b588c06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.129770 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 30928cce8b6f4dd1a0e2a5480e488e24 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=1b4f1f3f44114f5b8700b7074f2f63d1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.130002 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30928cce8b6f4dd1a0e2a5480e488e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.131551 9124 tablet_bootstrap.cc:492] T 27c905e41bfb4d99af6491331b588c06 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:58.132447 30257 tablet_service.cc:1467] Processing CreateTablet for tablet 30928cce8b6f4dd1a0e2a5480e488e24 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=1b4f1f3f44114f5b8700b7074f2f63d1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.132742 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30928cce8b6f4dd1a0e2a5480e488e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.132705 32533 tablet_service.cc:1467] Processing CreateTablet for tablet 27c905e41bfb4d99af6491331b588c06 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=1b4f1f3f44114f5b8700b7074f2f63d1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.132946 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c905e41bfb4d99af6491331b588c06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.135298 30126 tablet_service.cc:1467] Processing CreateTablet for tablet efeb0f330ef04192999e217612c4461f (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=1b4f1f3f44114f5b8700b7074f2f63d1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.135434 329 tablet_service.cc:1467] Processing CreateTablet for tablet 27c905e41bfb4d99af6491331b588c06 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=1b4f1f3f44114f5b8700b7074f2f63d1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.135558 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efeb0f330ef04192999e217612c4461f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.135680 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c905e41bfb4d99af6491331b588c06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.138198 327 tablet_service.cc:1467] Processing CreateTablet for tablet 30928cce8b6f4dd1a0e2a5480e488e24 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=1b4f1f3f44114f5b8700b7074f2f63d1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.138456 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30928cce8b6f4dd1a0e2a5480e488e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.132447 30255 tablet_service.cc:1467] Processing CreateTablet for tablet efeb0f330ef04192999e217612c4461f (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=1b4f1f3f44114f5b8700b7074f2f63d1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.144613 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efeb0f330ef04192999e217612c4461f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.146621 9007 consensus_queue.cc:1035] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:58.152760 30014 raft_consensus.cc:1203] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:58.152845 9124 tablet_bootstrap.cc:654] T 27c905e41bfb4d99af6491331b588c06 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.152993 29993 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:58.156951 9126 tablet_bootstrap.cc:492] T 30928cce8b6f4dd1a0e2a5480e488e24 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:58.157248 29868 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:58.157397 9129 tablet_bootstrap.cc:492] T efeb0f330ef04192999e217612c4461f P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:58.157388 7853 consensus_queue.cc:1035] T 7534d5788f96488fbad5f6334f476224 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:58.158713 9128 tablet_bootstrap.cc:492] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:58.158501 9127 tablet_bootstrap.cc:492] T 27c905e41bfb4d99af6491331b588c06 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:58.163300 9126 tablet_bootstrap.cc:654] T 30928cce8b6f4dd1a0e2a5480e488e24 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.164115 9127 tablet_bootstrap.cc:654] T 27c905e41bfb4d99af6491331b588c06 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.164499 9128 tablet_bootstrap.cc:654] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.164749 9129 tablet_bootstrap.cc:654] T efeb0f330ef04192999e217612c4461f P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.168915 9128 tablet_bootstrap.cc:492] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:58.169062 9128 ts_tablet_manager.cc:1367] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:58.169348 30386 tablet_service.cc:1467] Processing CreateTablet for tablet d0127d86c0c941b6b3a83b1a1b58b8a8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ed8bffe17bd4463292c956eb017311b4]), partition=
I20240501 14:09:58.169581 30386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0127d86c0c941b6b3a83b1a1b58b8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.169703 9128 raft_consensus.cc:348] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:58.169868 9128 raft_consensus.cc:374] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.169936 9128 raft_consensus.cc:729] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.170130 9128 consensus_queue.cc:260] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:58.170539 9128 ts_tablet_manager.cc:1398] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:58.170744 9128 tablet_bootstrap.cc:492] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:58.171628 9128 tablet_bootstrap.cc:654] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.172585 30126 tablet_service.cc:1467] Processing CreateTablet for tablet d0127d86c0c941b6b3a83b1a1b58b8a8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ed8bffe17bd4463292c956eb017311b4]), partition=
I20240501 14:09:58.172822 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0127d86c0c941b6b3a83b1a1b58b8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.173019 9126 tablet_bootstrap.cc:492] T 30928cce8b6f4dd1a0e2a5480e488e24 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:58.173105 9126 ts_tablet_manager.cc:1367] T 30928cce8b6f4dd1a0e2a5480e488e24 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:09:58.175303 32533 tablet_service.cc:1467] Processing CreateTablet for tablet d0127d86c0c941b6b3a83b1a1b58b8a8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ed8bffe17bd4463292c956eb017311b4]), partition=
I20240501 14:09:58.175536 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0127d86c0c941b6b3a83b1a1b58b8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.172590 9129 tablet_bootstrap.cc:492] T efeb0f330ef04192999e217612c4461f P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:58.175772 9129 ts_tablet_manager.cc:1367] T efeb0f330ef04192999e217612c4461f P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 14:09:58.176419 9129 raft_consensus.cc:348] T efeb0f330ef04192999e217612c4461f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:58.176640 9129 raft_consensus.cc:374] T efeb0f330ef04192999e217612c4461f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.176735 9129 raft_consensus.cc:729] T efeb0f330ef04192999e217612c4461f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.176900 9129 consensus_queue.cc:260] T efeb0f330ef04192999e217612c4461f P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:58.177280 9129 ts_tablet_manager.cc:1398] T efeb0f330ef04192999e217612c4461f P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:58.178258 9126 raft_consensus.cc:348] T 30928cce8b6f4dd1a0e2a5480e488e24 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:58.178395 9126 raft_consensus.cc:374] T 30928cce8b6f4dd1a0e2a5480e488e24 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.178429 9126 raft_consensus.cc:729] T 30928cce8b6f4dd1a0e2a5480e488e24 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.178591 9126 consensus_queue.cc:260] T 30928cce8b6f4dd1a0e2a5480e488e24 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:58.179003 9126 ts_tablet_manager.cc:1398] T 30928cce8b6f4dd1a0e2a5480e488e24 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:58.180452 9129 tablet_bootstrap.cc:492] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:58.181587 9129 tablet_bootstrap.cc:654] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.184435 9124 tablet_bootstrap.cc:492] T 27c905e41bfb4d99af6491331b588c06 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:58.184576 9124 ts_tablet_manager.cc:1367] T 27c905e41bfb4d99af6491331b588c06 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.000s
I20240501 14:09:58.185031 9124 raft_consensus.cc:348] T 27c905e41bfb4d99af6491331b588c06 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:58.185246 9124 raft_consensus.cc:374] T 27c905e41bfb4d99af6491331b588c06 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.185312 9124 raft_consensus.cc:729] T 27c905e41bfb4d99af6491331b588c06 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.185477 9124 consensus_queue.cc:260] T 27c905e41bfb4d99af6491331b588c06 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:58.188079 9124 ts_tablet_manager.cc:1398] T 27c905e41bfb4d99af6491331b588c06 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:58.188344 9124 tablet_bootstrap.cc:492] T 30928cce8b6f4dd1a0e2a5480e488e24 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:58.189833 9124 tablet_bootstrap.cc:654] T 30928cce8b6f4dd1a0e2a5480e488e24 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.190565 9126 tablet_bootstrap.cc:492] T efeb0f330ef04192999e217612c4461f P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:58.191525 9126 tablet_bootstrap.cc:654] T efeb0f330ef04192999e217612c4461f P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.193032 9127 tablet_bootstrap.cc:492] T 27c905e41bfb4d99af6491331b588c06 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:58.193487 9127 ts_tablet_manager.cc:1367] T 27c905e41bfb4d99af6491331b588c06 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20240501 14:09:58.195752 32664 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:58.196034 9127 raft_consensus.cc:348] T 27c905e41bfb4d99af6491331b588c06 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:58.196310 9127 raft_consensus.cc:374] T 27c905e41bfb4d99af6491331b588c06 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.196403 9127 raft_consensus.cc:729] T 27c905e41bfb4d99af6491331b588c06 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.196660 9127 consensus_queue.cc:260] T 27c905e41bfb4d99af6491331b588c06 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:58.199223 9127 ts_tablet_manager.cc:1398] T 27c905e41bfb4d99af6491331b588c06 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:58.202894 9127 tablet_bootstrap.cc:492] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:58.203881 9127 tablet_bootstrap.cc:654] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.205147 9128 tablet_bootstrap.cc:492] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:58.205309 9128 ts_tablet_manager.cc:1367] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20240501 14:09:58.206306 9128 raft_consensus.cc:348] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:58.206480 9128 raft_consensus.cc:374] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.206547 9128 raft_consensus.cc:729] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.206730 9128 consensus_queue.cc:260] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:58.215679 9129 tablet_bootstrap.cc:492] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:58.215852 9129 ts_tablet_manager.cc:1367] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20240501 14:09:58.216394 9129 raft_consensus.cc:348] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:58.216470 9137 tablet_bootstrap.cc:492] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:58.216544 9129 raft_consensus.cc:374] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.216609 9129 raft_consensus.cc:729] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.216759 9129 consensus_queue.cc:260] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:58.216861 9125 tablet_bootstrap.cc:492] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:58.217682 9137 tablet_bootstrap.cc:654] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.217744 9128 ts_tablet_manager.cc:1398] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240501 14:09:58.217957 9129 ts_tablet_manager.cc:1398] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:58.223882 9126 tablet_bootstrap.cc:492] T efeb0f330ef04192999e217612c4461f P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:58.223982 9126 ts_tablet_manager.cc:1367] T efeb0f330ef04192999e217612c4461f P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.000s
I20240501 14:09:58.224521 9126 raft_consensus.cc:348] T efeb0f330ef04192999e217612c4461f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:58.224635 9126 raft_consensus.cc:374] T efeb0f330ef04192999e217612c4461f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.224669 9126 raft_consensus.cc:729] T efeb0f330ef04192999e217612c4461f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.224822 9126 consensus_queue.cc:260] T efeb0f330ef04192999e217612c4461f P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:58.225100 9126 ts_tablet_manager.cc:1398] T efeb0f330ef04192999e217612c4461f P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:58.225111 9124 tablet_bootstrap.cc:492] T 30928cce8b6f4dd1a0e2a5480e488e24 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:58.225191 9124 ts_tablet_manager.cc:1367] T 30928cce8b6f4dd1a0e2a5480e488e24 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.002s
I20240501 14:09:58.225646 9124 raft_consensus.cc:348] T 30928cce8b6f4dd1a0e2a5480e488e24 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:58.225755 9124 raft_consensus.cc:374] T 30928cce8b6f4dd1a0e2a5480e488e24 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.225791 9124 raft_consensus.cc:729] T 30928cce8b6f4dd1a0e2a5480e488e24 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.225917 9124 consensus_queue.cc:260] T 30928cce8b6f4dd1a0e2a5480e488e24 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:58.226141 8595 raft_consensus.cc:482] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:58.226213 8595 raft_consensus.cc:504] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:58.226583 8595 leader_election.cc:288] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:58.227195 32552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c905e41bfb4d99af6491331b588c06" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:58.227334 32552 raft_consensus.cc:2454] T 27c905e41bfb4d99af6491331b588c06 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:58.227694 32753 leader_election.cc:302] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:58.228096 9124 ts_tablet_manager.cc:1398] T 30928cce8b6f4dd1a0e2a5480e488e24 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:58.228483 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c905e41bfb4d99af6491331b588c06" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:58.228952 32683 raft_consensus.cc:2454] T 27c905e41bfb4d99af6491331b588c06 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:58.230583 8595 raft_consensus.cc:2781] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:58.230638 8595 raft_consensus.cc:482] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:58.230671 8595 raft_consensus.cc:3037] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.231734 8595 raft_consensus.cc:504] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:58.232035 8595 leader_election.cc:288] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:09:58.232589 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c905e41bfb4d99af6491331b588c06" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:58.232678 32683 raft_consensus.cc:3037] T 27c905e41bfb4d99af6491331b588c06 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.233543 32683 raft_consensus.cc:2454] T 27c905e41bfb4d99af6491331b588c06 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:58.233906 32750 leader_election.cc:302] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:58.233937 32552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c905e41bfb4d99af6491331b588c06" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:58.234071 32552 raft_consensus.cc:3037] T 27c905e41bfb4d99af6491331b588c06 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.234977 32552 raft_consensus.cc:2454] T 27c905e41bfb4d99af6491331b588c06 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:58.235045 8595 raft_consensus.cc:2781] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:58.235148 8595 raft_consensus.cc:686] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:58.235282 8595 consensus_queue.cc:237] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:58.237821 29868 catalog_manager.cc:5484] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:58.241027 9125 tablet_bootstrap.cc:654] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.242164 9127 tablet_bootstrap.cc:492] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:58.242168 9125 tablet_bootstrap.cc:492] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:58.242247 9125 ts_tablet_manager.cc:1367] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240501 14:09:58.242259 9127 ts_tablet_manager.cc:1367] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.000s
I20240501 14:09:58.242655 9125 raft_consensus.cc:348] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:58.242753 9125 raft_consensus.cc:374] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.242702 9127 raft_consensus.cc:348] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:58.242779 9125 raft_consensus.cc:729] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.242790 9127 raft_consensus.cc:374] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.242820 9127 raft_consensus.cc:729] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.242882 9125 consensus_queue.cc:260] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:58.242924 9127 consensus_queue.cc:260] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:58.243127 9125 ts_tablet_manager.cc:1398] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:58.243196 9127 ts_tablet_manager.cc:1398] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:58.249866 9137 tablet_bootstrap.cc:492] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:58.249984 9137 ts_tablet_manager.cc:1367] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.000s
I20240501 14:09:58.250486 9137 raft_consensus.cc:348] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:58.250607 9137 raft_consensus.cc:374] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.250692 9137 raft_consensus.cc:729] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.250823 9137 consensus_queue.cc:260] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:58.251163 9137 ts_tablet_manager.cc:1398] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:58.265197 6185 raft_consensus.cc:482] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:58.265291 6185 raft_consensus.cc:504] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:58.265650 6185 leader_election.cc:288] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:58.265937 32552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0127d86c0c941b6b3a83b1a1b58b8a8" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:09:58.266074 32552 raft_consensus.cc:2454] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:58.266042 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0127d86c0c941b6b3a83b1a1b58b8a8" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:58.266160 30145 raft_consensus.cc:2454] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:58.266428 30344 leader_election.cc:302] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:58.266737 6185 raft_consensus.cc:2781] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:58.266803 6185 raft_consensus.cc:482] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:58.266844 6185 raft_consensus.cc:3037] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.267975 6185 raft_consensus.cc:504] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:58.268309 6185 leader_election.cc:288] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:09:58.268579 32552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0127d86c0c941b6b3a83b1a1b58b8a8" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:09:58.268709 32552 raft_consensus.cc:3037] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.268846 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0127d86c0c941b6b3a83b1a1b58b8a8" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:58.268940 30145 raft_consensus.cc:3037] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.269760 32552 raft_consensus.cc:2454] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:58.269994 30145 raft_consensus.cc:2454] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:58.270125 30344 leader_election.cc:302] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:58.270391 6185 raft_consensus.cc:2781] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:58.270545 6185 raft_consensus.cc:686] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:58.270725 6185 consensus_queue.cc:237] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:09:58.272974 29872 catalog_manager.cc:5484] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:09:58.278883 349 consensus_queue.cc:237] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:09:58.279987 32416 raft_consensus.cc:1261] T bf0712a5e11c46a291b90bfb491b8b25 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:58.280351 32552 raft_consensus.cc:1261] T bf0712a5e11c46a291b90bfb491b8b25 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:58.280351 8595 consensus_queue.cc:1035] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:58.280722 8595 consensus_queue.cc:1035] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:58.282232 8595 raft_consensus.cc:2932] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:58.282967 32552 raft_consensus.cc:2932] T bf0712a5e11c46a291b90bfb491b8b25 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:58.283325 32416 raft_consensus.cc:2932] T bf0712a5e11c46a291b90bfb491b8b25 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:58.286294 29868 catalog_manager.cc:5484] T bf0712a5e11c46a291b90bfb491b8b25 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:09:58.290630 32664 tablet_service.cc:1514] Processing DeleteTablet for tablet bf0712a5e11c46a291b90bfb491b8b25 with delete_type TABLET_DATA_TOMBSTONED (TS db935501e9104de5b15c8780681d6c38 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35078
I20240501 14:09:58.290952 9165 tablet_replica.cc:330] T bf0712a5e11c46a291b90bfb491b8b25 P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:09:58.291111 9165 raft_consensus.cc:2229] T bf0712a5e11c46a291b90bfb491b8b25 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:58.291244 9165 raft_consensus.cc:2258] T bf0712a5e11c46a291b90bfb491b8b25 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:58.291844 9165 ts_tablet_manager.cc:1875] T bf0712a5e11c46a291b90bfb491b8b25 P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:58.294586 9165 ts_tablet_manager.cc:1888] T bf0712a5e11c46a291b90bfb491b8b25 P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:09:58.294704 9165 log.cc:1196] T bf0712a5e11c46a291b90bfb491b8b25 P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/bf0712a5e11c46a291b90bfb491b8b25
I20240501 14:09:58.295214 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet bf0712a5e11c46a291b90bfb491b8b25 (table rebalancer_extra_table_26 [id=fa42a548a7f24eab9694087203fda9ba]) successfully deleted
I20240501 14:09:58.330174 7382 raft_consensus.cc:482] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:58.330302 7382 raft_consensus.cc:504] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:58.330718 7382 leader_election.cc:288] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:58.331120 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30928cce8b6f4dd1a0e2a5480e488e24" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:58.331302 32683 raft_consensus.cc:2454] T 30928cce8b6f4dd1a0e2a5480e488e24 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:58.331542 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30928cce8b6f4dd1a0e2a5480e488e24" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:58.331606 32750 leader_election.cc:302] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:09:58.331686 30264 raft_consensus.cc:2454] T 30928cce8b6f4dd1a0e2a5480e488e24 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:58.331760 7382 raft_consensus.cc:2781] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:58.331806 7382 raft_consensus.cc:482] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:58.331841 7382 raft_consensus.cc:3037] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.333009 7382 raft_consensus.cc:504] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:58.333382 7382 leader_election.cc:288] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:58.333881 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30928cce8b6f4dd1a0e2a5480e488e24" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:09:58.333976 32684 raft_consensus.cc:3037] T 30928cce8b6f4dd1a0e2a5480e488e24 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.334450 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30928cce8b6f4dd1a0e2a5480e488e24" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:58.334556 30277 raft_consensus.cc:3037] T 30928cce8b6f4dd1a0e2a5480e488e24 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.335047 32684 raft_consensus.cc:2454] T 30928cce8b6f4dd1a0e2a5480e488e24 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:58.335336 30277 raft_consensus.cc:2454] T 30928cce8b6f4dd1a0e2a5480e488e24 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:58.335623 32752 leader_election.cc:302] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:09:58.335811 7382 raft_consensus.cc:2781] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:58.335883 7382 raft_consensus.cc:686] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:58.335997 7382 consensus_queue.cc:237] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:58.339560 29872 catalog_manager.cc:5484] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:58.375936 29866 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:58.380983 30145 raft_consensus.cc:1261] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:58.381425 8442 consensus_queue.cc:1035] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:58.383201 32552 raft_consensus.cc:1261] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:58.383867 8442 consensus_queue.cc:1035] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:58.386485 29866 catalog_manager.cc:5576] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457) does not have the latest schema for tablet d0127d86c0c941b6b3a83b1a1b58b8a8 (table rebalancer_dynamic_table [id=ed8bffe17bd4463292c956eb017311b4]). Expected version 1 got 0
I20240501 14:09:58.387317 29873 catalog_manager.cc:5576] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497) does not have the latest schema for tablet d0127d86c0c941b6b3a83b1a1b58b8a8 (table rebalancer_dynamic_table [id=ed8bffe17bd4463292c956eb017311b4]). Expected version 1 got 0
I20240501 14:09:58.390100 9016 tablet.cc:1722] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:58.389971 9213 tablet.cc:1722] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:58.390404 9016 tablet.cc:1620] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3: MemRowSet was empty: no flush needed.
I20240501 14:09:58.390424 9213 tablet.cc:1620] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7: MemRowSet was empty: no flush needed.
I20240501 14:09:58.393399 7999 raft_consensus.cc:482] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:58.393502 7999 raft_consensus.cc:504] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:58.393990 7999 leader_election.cc:288] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:58.394330 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efeb0f330ef04192999e217612c4461f" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:58.394465 30145 raft_consensus.cc:2454] T efeb0f330ef04192999e217612c4461f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:58.394454 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efeb0f330ef04192999e217612c4461f" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:58.394574 30277 raft_consensus.cc:2454] T efeb0f330ef04192999e217612c4461f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:09:58.394784 32359 leader_election.cc:302] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:58.395316 7999 raft_consensus.cc:2781] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:58.395380 7999 raft_consensus.cc:482] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:58.395426 7999 raft_consensus.cc:3037] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.395658 29866 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=ed8bffe17bd4463292c956eb017311b4] alter complete (version 1)
I20240501 14:09:58.396499 7999 raft_consensus.cc:504] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:58.396814 7999 leader_election.cc:288] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:58.397416 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efeb0f330ef04192999e217612c4461f" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:58.397516 30277 raft_consensus.cc:3037] T efeb0f330ef04192999e217612c4461f P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.397490 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efeb0f330ef04192999e217612c4461f" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:58.397578 30145 raft_consensus.cc:3037] T efeb0f330ef04192999e217612c4461f P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.398545 30277 raft_consensus.cc:2454] T efeb0f330ef04192999e217612c4461f P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:58.398545 30145 raft_consensus.cc:2454] T efeb0f330ef04192999e217612c4461f P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:09:58.398824 32358 leader_election.cc:302] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:58.398970 7999 raft_consensus.cc:2781] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:58.399060 7999 raft_consensus.cc:686] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:09:58.399180 7999 consensus_queue.cc:237] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:58.401577 29873 catalog_manager.cc:5484] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:58.404764 9008 tablet.cc:1722] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:58.405058 9008 tablet.cc:1620] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:58.406313 29873 catalog_manager.cc:5576] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497) does not have the latest schema for tablet d0127d86c0c941b6b3a83b1a1b58b8a8 (table rebalancer_dynamic_table [id=ed8bffe17bd4463292c956eb017311b4]). Expected version 1 got 0
I20240501 14:09:58.439254 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:58.439468 32533 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:58.456959 30126 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:09:58.458909 29993 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:58.465139 32400 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:09:58.475975 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:09:58.476192 30386 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:58.484400 32664 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:58.497107 29871 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:58.505038 9213 tablet.cc:1722] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:58.505347 9213 tablet.cc:1620] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7: MemRowSet was empty: no flush needed.
I20240501 14:09:58.506451 9016 tablet.cc:1722] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:58.506718 9016 tablet.cc:1620] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3: MemRowSet was empty: no flush needed.
I20240501 14:09:58.513051 9008 tablet.cc:1722] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:58.513334 9008 tablet.cc:1620] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:58.518613 29870 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=ed8bffe17bd4463292c956eb017311b4] alter complete (version 2)
I20240501 14:09:58.553385 32683 consensus_queue.cc:237] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:58.554531 30277 raft_consensus.cc:1261] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:58.554615 30145 raft_consensus.cc:1261] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:58.554870 6832 consensus_queue.cc:1035] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:58.555017 8512 consensus_queue.cc:1035] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:58.556677 8512 raft_consensus.cc:2932] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:58.557058 30277 raft_consensus.cc:2932] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:58.559692 30145 raft_consensus.cc:2932] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:58.560417 29870 catalog_manager.cc:5484] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: config changed from index 2 to 3, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:58.564522 30386 tablet_service.cc:1514] Processing DeleteTablet for tablet d8430bd8cb7549f0aa4ede9a3d1ffe6e with delete_type TABLET_DATA_TOMBSTONED (TS d1c217469ae4478ebd49e4ed9f7352d7 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:58.564946 9249 tablet_replica.cc:330] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:58.565094 9249 raft_consensus.cc:2229] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:58.565227 9249 raft_consensus.cc:2258] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:58.565858 9249 ts_tablet_manager.cc:1875] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:58.568117 9249 ts_tablet_manager.cc:1888] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:09:58.568276 9249 log.cc:1196] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/d8430bd8cb7549f0aa4ede9a3d1ffe6e
I20240501 14:09:58.569000 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet d8430bd8cb7549f0aa4ede9a3d1ffe6e (table rebalancer_extra_table_42 [id=5cc0b546e462440f9284a6628020d724]) successfully deleted
I20240501 14:09:58.622751 29871 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:58.633541 9213 tablet.cc:1722] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:58.636092 9016 tablet.cc:1722] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:58.637444 9008 tablet.cc:1722] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:58.654584 29871 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=ed8bffe17bd4463292c956eb017311b4] alter complete (version 3)
I20240501 14:09:58.659307 7382 consensus_queue.cc:1035] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:58.676496 8595 consensus_queue.cc:1035] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:58.714751 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:58.716652 30386 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:58.719442 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:09:58.719704 32400 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:09:58.722997 29993 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:58.741801 30126 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:09:58.744691 32533 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:58.756187 7382 consensus_queue.cc:1035] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:58.756435 29873 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:58.756568 29873 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:58.757723 32664 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:58.758967 29873 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet d0127d86c0c941b6b3a83b1a1b58b8a8
I20240501 14:09:58.759893 32533 tablet_service.cc:1514] Processing DeleteTablet for tablet d0127d86c0c941b6b3a83b1a1b58b8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:58 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:58.760432 30126 tablet_service.cc:1514] Processing DeleteTablet for tablet d0127d86c0c941b6b3a83b1a1b58b8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:58 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:58.760708 9322 tablet_replica.cc:330] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:58.760875 9322 raft_consensus.cc:2229] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:58.761034 9322 raft_consensus.cc:2258] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:58.761662 9321 tablet_replica.cc:330] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:58.761837 9321 raft_consensus.cc:2229] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:58.761982 9321 raft_consensus.cc:2258] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:58.762589 9321 ts_tablet_manager.cc:1875] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:58.764961 9321 ts_tablet_manager.cc:1888] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:58.765108 9321 log.cc:1196] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/d0127d86c0c941b6b3a83b1a1b58b8a8
I20240501 14:09:58.765518 9321 ts_tablet_manager.cc:1909] T d0127d86c0c941b6b3a83b1a1b58b8a8 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:58.766212 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet d0127d86c0c941b6b3a83b1a1b58b8a8 (table rebalancer_dynamic_table_new_name [id=ed8bffe17bd4463292c956eb017311b4]) successfully deleted
I20240501 14:09:58.767434 30386 tablet_service.cc:1514] Processing DeleteTablet for tablet d0127d86c0c941b6b3a83b1a1b58b8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:58 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:58.767621 9249 tablet_replica.cc:330] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:58.767740 9249 raft_consensus.cc:2229] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:58.767915 9249 raft_consensus.cc:2258] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:58.768803 9249 ts_tablet_manager.cc:1875] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:58.770738 9249 ts_tablet_manager.cc:1888] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:58.770785 9249 log.cc:1196] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/d0127d86c0c941b6b3a83b1a1b58b8a8
I20240501 14:09:58.770982 9249 ts_tablet_manager.cc:1909] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:09:58.772598 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet d0127d86c0c941b6b3a83b1a1b58b8a8 (table rebalancer_dynamic_table_new_name [id=ed8bffe17bd4463292c956eb017311b4]) successfully deleted
I20240501 14:09:58.776592 9322 ts_tablet_manager.cc:1875] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:58.778900 9322 ts_tablet_manager.cc:1888] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:58.779026 9322 log.cc:1196] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/d0127d86c0c941b6b3a83b1a1b58b8a8
I20240501 14:09:58.781083 9322 ts_tablet_manager.cc:1909] T d0127d86c0c941b6b3a83b1a1b58b8a8 P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:09:58.784864 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet d0127d86c0c941b6b3a83b1a1b58b8a8 (table rebalancer_dynamic_table_new_name [id=ed8bffe17bd4463292c956eb017311b4]) successfully deleted
I20240501 14:09:58.787547 8595 consensus_queue.cc:1035] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:58.797108 7999 consensus_queue.cc:1035] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:58.798017 7305 consensus_queue.cc:1035] T efeb0f330ef04192999e217612c4461f P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:58.852499 32683 consensus_queue.cc:237] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:58.853734 30277 raft_consensus.cc:1261] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:58.853734 349 raft_consensus.cc:1261] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:58.854089 8512 consensus_queue.cc:1035] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:58.854235 9244 consensus_queue.cc:1035] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:58.855867 6832 raft_consensus.cc:2932] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:58.856364 30277 raft_consensus.cc:2932] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:58.856387 349 raft_consensus.cc:2932] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:58.859675 29873 catalog_manager.cc:5484] T 7e337eb723974834b07aac5eacf73232 P db935501e9104de5b15c8780681d6c38 reported cstate change: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:09:58.861534 29864 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:58.863201 32533 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e337eb723974834b07aac5eacf73232 with delete_type TABLET_DATA_TOMBSTONED (TS 5f8a0cfdfdca4148a75686dc88059cf3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:58.863415 9321 tablet_replica.cc:330] T 7e337eb723974834b07aac5eacf73232 P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:58.863559 9321 raft_consensus.cc:2229] T 7e337eb723974834b07aac5eacf73232 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:58.863679 9321 raft_consensus.cc:2258] T 7e337eb723974834b07aac5eacf73232 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:58.864204 9321 ts_tablet_manager.cc:1875] T 7e337eb723974834b07aac5eacf73232 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:58.867322 9321 ts_tablet_manager.cc:1888] T 7e337eb723974834b07aac5eacf73232 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:09:58.867393 9321 log.cc:1196] T 7e337eb723974834b07aac5eacf73232 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/7e337eb723974834b07aac5eacf73232
I20240501 14:09:58.867944 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet 7e337eb723974834b07aac5eacf73232 (table rebalancer_extra_table_34 [id=c0df7ced31ed4653bf4f71f2bb576ed9]) successfully deleted
I20240501 14:09:58.869369 32533 tablet_service.cc:1467] Processing CreateTablet for tablet 424b43f9d6f844e79d3135d80b5a17ec (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6606b1cbd31447dfb28d0713ab5cb020]), partition=
I20240501 14:09:58.869604 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 424b43f9d6f844e79d3135d80b5a17ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.869601 327 tablet_service.cc:1467] Processing CreateTablet for tablet 424b43f9d6f844e79d3135d80b5a17ec (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6606b1cbd31447dfb28d0713ab5cb020]), partition=
I20240501 14:09:58.869822 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 424b43f9d6f844e79d3135d80b5a17ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.871590 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 424b43f9d6f844e79d3135d80b5a17ec (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6606b1cbd31447dfb28d0713ab5cb020]), partition=
I20240501 14:09:58.871721 9352 tablet_bootstrap.cc:492] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:58.871815 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 424b43f9d6f844e79d3135d80b5a17ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.872555 9352 tablet_bootstrap.cc:654] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.873677 9352 tablet_bootstrap.cc:492] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:58.873790 9352 ts_tablet_manager.cc:1367] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:58.874004 9351 tablet_bootstrap.cc:492] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:58.874230 9354 tablet_bootstrap.cc:492] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:58.874307 9352 raft_consensus.cc:348] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:58.874444 9352 raft_consensus.cc:374] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.874476 9352 raft_consensus.cc:729] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.874583 9352 consensus_queue.cc:260] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:58.874857 9352 ts_tablet_manager.cc:1398] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:58.874922 9351 tablet_bootstrap.cc:654] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.875087 9354 tablet_bootstrap.cc:654] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.876361 9351 tablet_bootstrap.cc:492] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:58.876448 9351 ts_tablet_manager.cc:1367] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:58.876665 9354 tablet_bootstrap.cc:492] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:58.876752 9354 ts_tablet_manager.cc:1367] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:58.876886 9351 raft_consensus.cc:348] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:58.876989 9351 raft_consensus.cc:374] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.877041 9351 raft_consensus.cc:729] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.877149 9351 consensus_queue.cc:260] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:58.877179 9354 raft_consensus.cc:348] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:58.877285 9354 raft_consensus.cc:374] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.877315 9354 raft_consensus.cc:729] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.877415 9351 ts_tablet_manager.cc:1398] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:58.877418 9354 consensus_queue.cc:260] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:58.877682 9354 ts_tablet_manager.cc:1398] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:58.910871 29864 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:58.919034 30255 tablet_service.cc:1467] Processing CreateTablet for tablet f7264487a83e4574a7946e53f5dc8612 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=8a43630f2be44f81a8a03ec942194486]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.919139 32400 tablet_service.cc:1467] Processing CreateTablet for tablet b816de53d6a74e798a3b4d8988165c54 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=8a43630f2be44f81a8a03ec942194486]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.919344 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7264487a83e4574a7946e53f5dc8612. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.919417 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b816de53d6a74e798a3b4d8988165c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.919678 30386 tablet_service.cc:1467] Processing CreateTablet for tablet 60090b33d0e24ff18fd6c73548b05a13 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=8a43630f2be44f81a8a03ec942194486]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.919926 30386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60090b33d0e24ff18fd6c73548b05a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.921525 9394 tablet_bootstrap.cc:492] T f7264487a83e4574a7946e53f5dc8612 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:58.921679 30257 tablet_service.cc:1467] Processing CreateTablet for tablet b816de53d6a74e798a3b4d8988165c54 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=8a43630f2be44f81a8a03ec942194486]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.921743 9354 tablet_bootstrap.cc:492] T b816de53d6a74e798a3b4d8988165c54 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:58.921873 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b816de53d6a74e798a3b4d8988165c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.922333 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 60090b33d0e24ff18fd6c73548b05a13 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=8a43630f2be44f81a8a03ec942194486]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.922482 9394 tablet_bootstrap.cc:654] T f7264487a83e4574a7946e53f5dc8612 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.922609 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60090b33d0e24ff18fd6c73548b05a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.922626 9354 tablet_bootstrap.cc:654] T b816de53d6a74e798a3b4d8988165c54 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.923993 9354 tablet_bootstrap.cc:492] T b816de53d6a74e798a3b4d8988165c54 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:58.924077 9354 ts_tablet_manager.cc:1367] T b816de53d6a74e798a3b4d8988165c54 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:58.924551 9354 raft_consensus.cc:348] T b816de53d6a74e798a3b4d8988165c54 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:58.924675 9354 raft_consensus.cc:374] T b816de53d6a74e798a3b4d8988165c54 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.924712 9354 raft_consensus.cc:729] T b816de53d6a74e798a3b4d8988165c54 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.924826 9354 consensus_queue.cc:260] T b816de53d6a74e798a3b4d8988165c54 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:58.925175 30388 tablet_service.cc:1467] Processing CreateTablet for tablet f7264487a83e4574a7946e53f5dc8612 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=8a43630f2be44f81a8a03ec942194486]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.925223 32534 tablet_service.cc:1467] Processing CreateTablet for tablet b816de53d6a74e798a3b4d8988165c54 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=8a43630f2be44f81a8a03ec942194486]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.925407 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7264487a83e4574a7946e53f5dc8612. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.925499 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b816de53d6a74e798a3b4d8988165c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.926401 9395 tablet_bootstrap.cc:492] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:58.927361 9395 tablet_bootstrap.cc:654] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.927682 9351 tablet_bootstrap.cc:492] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:58.928632 32533 tablet_service.cc:1467] Processing CreateTablet for tablet f7264487a83e4574a7946e53f5dc8612 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=8a43630f2be44f81a8a03ec942194486]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.928843 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7264487a83e4574a7946e53f5dc8612. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.929052 9351 tablet_bootstrap.cc:654] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.929709 9354 ts_tablet_manager.cc:1398] T b816de53d6a74e798a3b4d8988165c54 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:58.930095 9351 tablet_bootstrap.cc:492] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:58.930171 9351 ts_tablet_manager.cc:1367] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:58.930521 9351 raft_consensus.cc:348] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:58.930591 9351 raft_consensus.cc:374] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.930611 9351 raft_consensus.cc:729] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.930694 9351 consensus_queue.cc:260] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:58.930946 9351 ts_tablet_manager.cc:1398] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:58.931196 9351 tablet_bootstrap.cc:492] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:09:58.933187 30126 tablet_service.cc:1467] Processing CreateTablet for tablet 60090b33d0e24ff18fd6c73548b05a13 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=8a43630f2be44f81a8a03ec942194486]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.933468 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60090b33d0e24ff18fd6c73548b05a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.938179 9395 tablet_bootstrap.cc:492] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:58.938283 9395 ts_tablet_manager.cc:1367] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:58.938927 9395 raft_consensus.cc:348] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:58.939041 9395 raft_consensus.cc:374] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.939076 9395 raft_consensus.cc:729] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.939180 9395 consensus_queue.cc:260] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:58.939472 9395 ts_tablet_manager.cc:1398] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:58.939579 9351 tablet_bootstrap.cc:654] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.939859 9395 tablet_bootstrap.cc:492] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:58.940639 9351 tablet_bootstrap.cc:492] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:09:58.940716 9351 ts_tablet_manager.cc:1367] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 14:09:58.941007 9395 tablet_bootstrap.cc:654] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.941154 9351 raft_consensus.cc:348] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:58.941260 9351 raft_consensus.cc:374] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.941311 9351 raft_consensus.cc:729] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.941439 9351 consensus_queue.cc:260] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:58.941725 9351 ts_tablet_manager.cc:1398] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:58.943354 9402 tablet_bootstrap.cc:492] T 60090b33d0e24ff18fd6c73548b05a13 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:58.944406 9402 tablet_bootstrap.cc:654] T 60090b33d0e24ff18fd6c73548b05a13 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.945183 9398 tablet_bootstrap.cc:492] T 60090b33d0e24ff18fd6c73548b05a13 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:58.945552 9395 tablet_bootstrap.cc:492] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:58.945662 9395 ts_tablet_manager.cc:1367] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:58.946218 9398 tablet_bootstrap.cc:654] T 60090b33d0e24ff18fd6c73548b05a13 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.946511 9395 raft_consensus.cc:348] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:58.946934 9402 tablet_bootstrap.cc:492] T 60090b33d0e24ff18fd6c73548b05a13 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:58.947012 9402 ts_tablet_manager.cc:1367] T 60090b33d0e24ff18fd6c73548b05a13 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:09:58.947501 9402 raft_consensus.cc:348] T 60090b33d0e24ff18fd6c73548b05a13 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:58.947588 9402 raft_consensus.cc:374] T 60090b33d0e24ff18fd6c73548b05a13 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.947616 9402 raft_consensus.cc:729] T 60090b33d0e24ff18fd6c73548b05a13 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.947718 9402 consensus_queue.cc:260] T 60090b33d0e24ff18fd6c73548b05a13 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:58.947971 9402 ts_tablet_manager.cc:1398] T 60090b33d0e24ff18fd6c73548b05a13 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:58.946635 9395 raft_consensus.cc:374] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.948863 9395 raft_consensus.cc:729] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.948971 9395 consensus_queue.cc:260] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:58.949821 9395 ts_tablet_manager.cc:1398] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:58.950297 7853 raft_consensus.cc:482] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:58.950371 7853 raft_consensus.cc:504] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:58.950682 7853 leader_election.cc:288] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:58.951182 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "424b43f9d6f844e79d3135d80b5a17ec" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:58.951290 32416 raft_consensus.cc:2454] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:58.951715 32489 leader_election.cc:302] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:58.951920 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "424b43f9d6f844e79d3135d80b5a17ec" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:09:58.952049 349 raft_consensus.cc:2454] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:58.952435 7853 raft_consensus.cc:2781] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:58.952489 7853 raft_consensus.cc:482] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:58.952528 7853 raft_consensus.cc:3037] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.953413 9394 tablet_bootstrap.cc:492] T f7264487a83e4574a7946e53f5dc8612 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:58.953449 7853 raft_consensus.cc:504] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:58.953501 9394 ts_tablet_manager.cc:1367] T f7264487a83e4574a7946e53f5dc8612 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20240501 14:09:58.953775 7853 leader_election.cc:288] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:09:58.955708 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "424b43f9d6f844e79d3135d80b5a17ec" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:58.955695 9394 raft_consensus.cc:348] T f7264487a83e4574a7946e53f5dc8612 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:58.955789 9394 raft_consensus.cc:374] T f7264487a83e4574a7946e53f5dc8612 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.955812 32416 raft_consensus.cc:3037] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.955821 9394 raft_consensus.cc:729] T f7264487a83e4574a7946e53f5dc8612 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.955929 9394 consensus_queue.cc:260] T f7264487a83e4574a7946e53f5dc8612 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:58.956178 9394 ts_tablet_manager.cc:1398] T f7264487a83e4574a7946e53f5dc8612 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240501 14:09:58.956351 9394 tablet_bootstrap.cc:492] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:58.956845 32416 raft_consensus.cc:2454] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:58.957304 9394 tablet_bootstrap.cc:654] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.958637 32489 leader_election.cc:302] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:58.959059 9394 tablet_bootstrap.cc:492] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:58.959137 9394 ts_tablet_manager.cc:1367] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:58.959167 9398 tablet_bootstrap.cc:492] T 60090b33d0e24ff18fd6c73548b05a13 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:58.959260 9398 ts_tablet_manager.cc:1367] T 60090b33d0e24ff18fd6c73548b05a13 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:09:58.959559 9394 raft_consensus.cc:348] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:58.959643 9394 raft_consensus.cc:374] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.959712 9394 raft_consensus.cc:729] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.959730 9398 raft_consensus.cc:348] T 60090b33d0e24ff18fd6c73548b05a13 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:58.959913 9398 raft_consensus.cc:374] T 60090b33d0e24ff18fd6c73548b05a13 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.959885 9394 consensus_queue.cc:260] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:58.960006 9398 raft_consensus.cc:729] T 60090b33d0e24ff18fd6c73548b05a13 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.960162 9398 consensus_queue.cc:260] T 60090b33d0e24ff18fd6c73548b05a13 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:58.960512 9398 ts_tablet_manager.cc:1398] T 60090b33d0e24ff18fd6c73548b05a13 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:58.960888 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "424b43f9d6f844e79d3135d80b5a17ec" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:09:58.960994 349 raft_consensus.cc:3037] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.961849 349 raft_consensus.cc:2454] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:58.962433 7853 raft_consensus.cc:2781] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:58.962512 7853 raft_consensus.cc:686] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:09:58.962620 7853 consensus_queue.cc:237] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:09:58.965286 29873 catalog_manager.cc:5484] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:09:58.967444 9394 ts_tablet_manager.cc:1398] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240501 14:09:58.999914 9007 raft_consensus.cc:482] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:59.000032 9007 raft_consensus.cc:504] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.000418 9007 leader_election.cc:288] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:59.001058 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b816de53d6a74e798a3b4d8988165c54" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:59.001173 32416 raft_consensus.cc:2454] T b816de53d6a74e798a3b4d8988165c54 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:59.001518 32489 leader_election.cc:302] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:59.002034 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b816de53d6a74e798a3b4d8988165c54" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:59.002147 30277 raft_consensus.cc:2454] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:59.002373 9007 raft_consensus.cc:2781] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:59.002429 9007 raft_consensus.cc:482] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:59.002466 9007 raft_consensus.cc:3037] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.003597 9007 raft_consensus.cc:504] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.003901 9007 leader_election.cc:288] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:59.004413 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b816de53d6a74e798a3b4d8988165c54" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:59.004515 32416 raft_consensus.cc:3037] T b816de53d6a74e798a3b4d8988165c54 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.005518 32416 raft_consensus.cc:2454] T b816de53d6a74e798a3b4d8988165c54 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:59.007172 32489 leader_election.cc:302] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:59.008399 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b816de53d6a74e798a3b4d8988165c54" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:59.008498 30277 raft_consensus.cc:3037] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.009465 30277 raft_consensus.cc:2454] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:59.015832 9007 raft_consensus.cc:2781] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:59.015913 9007 raft_consensus.cc:686] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:09:59.016033 9007 consensus_queue.cc:237] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.019450 29868 catalog_manager.cc:5484] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:59.030501 9007 raft_consensus.cc:482] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:59.030611 9007 raft_consensus.cc:504] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:59.031005 9007 leader_election.cc:288] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:59.031316 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7264487a83e4574a7946e53f5dc8612" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:59.031455 30277 raft_consensus.cc:2454] T f7264487a83e4574a7946e53f5dc8612 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:59.031423 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7264487a83e4574a7946e53f5dc8612" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:59.031533 30406 raft_consensus.cc:2454] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:09:59.031702 32489 leader_election.cc:302] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 5f8a0cfdfdca4148a75686dc88059cf3; no voters:
I20240501 14:09:59.031912 9007 raft_consensus.cc:2781] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:59.031970 9007 raft_consensus.cc:482] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:59.032006 9007 raft_consensus.cc:3037] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.033078 9007 raft_consensus.cc:504] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:59.033432 9007 leader_election.cc:288] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:09:59.033766 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7264487a83e4574a7946e53f5dc8612" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:59.033879 30406 raft_consensus.cc:3037] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.034101 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7264487a83e4574a7946e53f5dc8612" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:09:59.034184 30277 raft_consensus.cc:3037] T f7264487a83e4574a7946e53f5dc8612 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.034933 30406 raft_consensus.cc:2454] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:59.035241 30277 raft_consensus.cc:2454] T f7264487a83e4574a7946e53f5dc8612 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:09:59.035449 32487 leader_election.cc:302] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:59.035619 7853 raft_consensus.cc:2781] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:59.035715 7853 raft_consensus.cc:686] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:09:59.035833 7853 consensus_queue.cc:237] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:09:59.038408 29865 catalog_manager.cc:5484] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:09:59.089157 29865 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:59.092552 349 raft_consensus.cc:1261] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:59.092605 32416 raft_consensus.cc:1261] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:59.093032 9007 consensus_queue.cc:1035] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:59.093204 7853 consensus_queue.cc:1035] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:59.095901 9008 tablet.cc:1722] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:59.096216 9008 tablet.cc:1620] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:59.096956 29868 catalog_manager.cc:5576] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937) does not have the latest schema for tablet 424b43f9d6f844e79d3135d80b5a17ec (table rebalancer_dynamic_table [id=6606b1cbd31447dfb28d0713ab5cb020]). Expected version 1 got 0
I20240501 14:09:59.098698 29865 catalog_manager.cc:5576] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265) does not have the latest schema for tablet 424b43f9d6f844e79d3135d80b5a17ec (table rebalancer_dynamic_table [id=6606b1cbd31447dfb28d0713ab5cb020]). Expected version 1 got 0
I20240501 14:09:59.100226 9415 tablet.cc:1722] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:59.100653 9415 tablet.cc:1620] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:59.101478 9416 tablet.cc:1722] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:59.102036 9416 tablet.cc:1620] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:59.102695 29872 catalog_manager.cc:5576] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937) does not have the latest schema for tablet 424b43f9d6f844e79d3135d80b5a17ec (table rebalancer_dynamic_table [id=6606b1cbd31447dfb28d0713ab5cb020]). Expected version 1 got 0
I20240501 14:09:59.105435 29868 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=6606b1cbd31447dfb28d0713ab5cb020] alter complete (version 1)
I20240501 14:09:59.182240 6185 raft_consensus.cc:482] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:59.182374 6185 raft_consensus.cc:504] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:59.182770 6185 leader_election.cc:288] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:59.183099 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60090b33d0e24ff18fd6c73548b05a13" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:59.183262 30145 raft_consensus.cc:2454] T 60090b33d0e24ff18fd6c73548b05a13 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:59.183583 30344 leader_election.cc:302] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:59.183955 6185 raft_consensus.cc:2781] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:59.184015 6185 raft_consensus.cc:482] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:59.184051 6185 raft_consensus.cc:3037] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.184041 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60090b33d0e24ff18fd6c73548b05a13" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:09:59.184190 30014 raft_consensus.cc:2454] T 60090b33d0e24ff18fd6c73548b05a13 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:59.185179 6185 raft_consensus.cc:504] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:59.185525 6185 leader_election.cc:288] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:09:59.186105 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60090b33d0e24ff18fd6c73548b05a13" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:59.186223 30145 raft_consensus.cc:3037] T 60090b33d0e24ff18fd6c73548b05a13 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.187182 30145 raft_consensus.cc:2454] T 60090b33d0e24ff18fd6c73548b05a13 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:59.187741 30344 leader_election.cc:302] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:59.188066 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60090b33d0e24ff18fd6c73548b05a13" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:09:59.188190 30014 raft_consensus.cc:3037] T 60090b33d0e24ff18fd6c73548b05a13 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.188995 30014 raft_consensus.cc:2454] T 60090b33d0e24ff18fd6c73548b05a13 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:59.189369 6185 raft_consensus.cc:2781] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:59.189540 6185 raft_consensus.cc:686] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:59.189741 6185 consensus_queue.cc:237] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:59.192466 29865 catalog_manager.cc:5484] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:09:59.209471 29865 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:59.220186 9008 tablet.cc:1722] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:59.220571 9008 tablet.cc:1620] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3: MemRowSet was empty: no flush needed.
I20240501 14:09:59.221640 9416 tablet.cc:1722] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:59.221943 9416 tablet.cc:1620] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:59.222205 9415 tablet.cc:1722] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:59.222505 9415 tablet.cc:1620] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:59.233263 29993 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:59.233278 30257 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:59.233628 29865 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=6606b1cbd31447dfb28d0713ab5cb020] alter complete (version 2)
I20240501 14:09:59.241353 32533 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:59.243855 30388 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:59.251339 30126 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:09:59.267694 32664 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:59.277155 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:09:59.292423 32400 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:09:59.336305 30014 consensus_queue.cc:237] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:09:59.337393 349 raft_consensus.cc:1261] T af39c304693645269a6af59190e95cad P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:59.337405 32683 raft_consensus.cc:1261] T af39c304693645269a6af59190e95cad P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:59.337765 9288 consensus_queue.cc:1035] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:59.337980 7378 consensus_queue.cc:1035] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:59.339773 9452 raft_consensus.cc:2932] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:59.340139 32683 raft_consensus.cc:2932] T af39c304693645269a6af59190e95cad P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:59.340159 349 raft_consensus.cc:2932] T af39c304693645269a6af59190e95cad P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:59.341909 29865 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:59.343461 29868 catalog_manager.cc:5484] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e reported cstate change: config changed from index 6 to 7, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:09:59.350028 9008 tablet.cc:1722] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:59.350107 30388 tablet_service.cc:1514] Processing DeleteTablet for tablet af39c304693645269a6af59190e95cad with delete_type TABLET_DATA_TOMBSTONED (TS d1c217469ae4478ebd49e4ed9f7352d7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38118
I20240501 14:09:59.350440 9416 tablet.cc:1722] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:59.350867 9458 tablet_replica.cc:330] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:09:59.351035 9458 raft_consensus.cc:2229] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:59.351279 9458 raft_consensus.cc:2258] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:59.352504 9458 ts_tablet_manager.cc:1875] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:59.355177 9458 ts_tablet_manager.cc:1888] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:09:59.355290 9458 log.cc:1196] T af39c304693645269a6af59190e95cad P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/af39c304693645269a6af59190e95cad
I20240501 14:09:59.356088 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet af39c304693645269a6af59190e95cad (table rebalance_test_table_3 [id=b7b3f5bc4b6649c5a97ec9f02da4e967]) successfully deleted
I20240501 14:09:59.357367 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=6606b1cbd31447dfb28d0713ab5cb020] alter complete (version 3)
I20240501 14:09:59.357472 9415 tablet.cc:1722] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:59.428047 9412 consensus_queue.cc:1035] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:59.434033 9007 consensus_queue.cc:1035] T f7264487a83e4574a7946e53f5dc8612 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:59.443379 9007 consensus_queue.cc:1035] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:59.448336 7853 consensus_queue.cc:1035] T b816de53d6a74e798a3b4d8988165c54 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:59.464074 29869 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:59.464263 29869 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:59.466255 29869 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 424b43f9d6f844e79d3135d80b5a17ec
I20240501 14:09:59.466796 32533 tablet_service.cc:1514] Processing DeleteTablet for tablet 424b43f9d6f844e79d3135d80b5a17ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:59 UTC) from {username='slave'} at 127.0.0.1:47638
I20240501 14:09:59.467129 9504 tablet_replica.cc:330] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:09:59.467118 32400 tablet_service.cc:1514] Processing DeleteTablet for tablet 424b43f9d6f844e79d3135d80b5a17ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:59 UTC) from {username='slave'} at 127.0.0.1:53090
I20240501 14:09:59.467329 9504 raft_consensus.cc:2229] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:59.467346 327 tablet_service.cc:1514] Processing DeleteTablet for tablet 424b43f9d6f844e79d3135d80b5a17ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:59 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:09:59.467569 9504 raft_consensus.cc:2258] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:59.468173 9504 ts_tablet_manager.cc:1875] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:59.470247 9504 ts_tablet_manager.cc:1888] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:59.470366 9504 log.cc:1196] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/424b43f9d6f844e79d3135d80b5a17ec
I20240501 14:09:59.470147 9505 tablet_replica.cc:330] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:09:59.470667 9504 ts_tablet_manager.cc:1909] T 424b43f9d6f844e79d3135d80b5a17ec P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting consensus metadata
I20240501 14:09:59.471383 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet 424b43f9d6f844e79d3135d80b5a17ec (table rebalancer_dynamic_table_new_name [id=6606b1cbd31447dfb28d0713ab5cb020]) successfully deleted
I20240501 14:09:59.471621 9505 raft_consensus.cc:2229] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:59.471796 9505 raft_consensus.cc:2258] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:59.472504 9505 ts_tablet_manager.cc:1875] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:59.474253 9506 tablet_replica.cc:330] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:09:59.474433 9506 raft_consensus.cc:2229] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:59.475878 9506 raft_consensus.cc:2258] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:59.477798 9506 ts_tablet_manager.cc:1875] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:59.480358 9505 ts_tablet_manager.cc:1888] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:59.480470 9505 log.cc:1196] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/424b43f9d6f844e79d3135d80b5a17ec
I20240501 14:09:59.480773 9505 ts_tablet_manager.cc:1909] T 424b43f9d6f844e79d3135d80b5a17ec P a91029b90bf9415a89a2f70736759ad3: Deleting consensus metadata
I20240501 14:09:59.481482 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 424b43f9d6f844e79d3135d80b5a17ec (table rebalancer_dynamic_table_new_name [id=6606b1cbd31447dfb28d0713ab5cb020]) successfully deleted
I20240501 14:09:59.482271 9506 ts_tablet_manager.cc:1888] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:59.482380 9506 log.cc:1196] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/424b43f9d6f844e79d3135d80b5a17ec
I20240501 14:09:59.482673 9506 ts_tablet_manager.cc:1909] T 424b43f9d6f844e79d3135d80b5a17ec P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:09:59.483595 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 424b43f9d6f844e79d3135d80b5a17ec (table rebalancer_dynamic_table_new_name [id=6606b1cbd31447dfb28d0713ab5cb020]) successfully deleted
I20240501 14:09:59.496718 30257 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:59.499161 32533 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:59.504859 30388 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:59.506870 29993 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:59.511991 32400 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:09:59.513911 30126 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:09:59.518569 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:09:59.532955 32664 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:59.567821 29866 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:59.573823 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 8ecdda0ff32d4e0fac0b94588bb9d9e1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=51ac4aecbc2747528a1f54e5c7977ba7]), partition=
I20240501 14:09:59.573879 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 8ecdda0ff32d4e0fac0b94588bb9d9e1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=51ac4aecbc2747528a1f54e5c7977ba7]), partition=
I20240501 14:09:59.574105 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ecdda0ff32d4e0fac0b94588bb9d9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.574136 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ecdda0ff32d4e0fac0b94588bb9d9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.575883 9518 tablet_bootstrap.cc:492] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:59.576717 327 tablet_service.cc:1467] Processing CreateTablet for tablet 8ecdda0ff32d4e0fac0b94588bb9d9e1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=51ac4aecbc2747528a1f54e5c7977ba7]), partition=
I20240501 14:09:59.576943 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ecdda0ff32d4e0fac0b94588bb9d9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.577172 9518 tablet_bootstrap.cc:654] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.579146 9521 tablet_bootstrap.cc:492] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:59.580462 9519 tablet_bootstrap.cc:492] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:59.580549 9521 tablet_bootstrap.cc:654] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.581528 9521 tablet_bootstrap.cc:492] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:59.581691 9519 tablet_bootstrap.cc:654] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.581791 9521 ts_tablet_manager.cc:1367] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:59.582569 9521 raft_consensus.cc:348] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.582674 9521 raft_consensus.cc:374] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.582727 9521 raft_consensus.cc:729] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.582878 9521 consensus_queue.cc:260] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.582928 9519 tablet_bootstrap.cc:492] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:59.583009 9519 ts_tablet_manager.cc:1367] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:59.583175 9521 ts_tablet_manager.cc:1398] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:59.583452 9519 raft_consensus.cc:348] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.583635 9519 raft_consensus.cc:374] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.583709 9519 raft_consensus.cc:729] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.583891 9519 consensus_queue.cc:260] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.584203 9519 ts_tablet_manager.cc:1398] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:59.591589 9518 tablet_bootstrap.cc:492] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:59.591751 9518 ts_tablet_manager.cc:1367] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:09:59.594763 9518 raft_consensus.cc:348] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.594868 9518 raft_consensus.cc:374] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.594897 9518 raft_consensus.cc:729] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.595028 9518 consensus_queue.cc:260] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.595383 9518 ts_tablet_manager.cc:1398] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:59.616735 30406 consensus_queue.cc:237] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.618108 348 raft_consensus.cc:1261] T 940c8f54339f4a32be2942682a41c685 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:59.618273 32416 raft_consensus.cc:1261] T 940c8f54339f4a32be2942682a41c685 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:59.618764 8442 consensus_queue.cc:1035] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:59.619980 8442 consensus_queue.cc:1035] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:59.622056 8442 raft_consensus.cc:2932] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:59.625445 29866 catalog_manager.cc:5484] T 940c8f54339f4a32be2942682a41c685 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index 6 to 7, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:59.625870 32416 raft_consensus.cc:2932] T 940c8f54339f4a32be2942682a41c685 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:59.628993 30126 tablet_service.cc:1514] Processing DeleteTablet for tablet 940c8f54339f4a32be2942682a41c685 with delete_type TABLET_DATA_TOMBSTONED (TS d35c16dcb506426993811bb2519d7db3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45302
I20240501 14:09:59.629462 9548 tablet_replica.cc:330] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:09:59.629957 9548 raft_consensus.cc:2229] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:59.630216 9548 raft_consensus.cc:2258] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:59.631078 349 raft_consensus.cc:2932] T 940c8f54339f4a32be2942682a41c685 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:59.631604 9548 ts_tablet_manager.cc:1875] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:59.637624 9548 ts_tablet_manager.cc:1888] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:09:59.637758 9548 log.cc:1196] T 940c8f54339f4a32be2942682a41c685 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/940c8f54339f4a32be2942682a41c685
I20240501 14:09:59.639012 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 940c8f54339f4a32be2942682a41c685 (table rebalance_test_table_9 [id=abd0f09508ae4b6699a526d040fd6f96]) successfully deleted
I20240501 14:09:59.653797 7382 raft_consensus.cc:482] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:59.653890 7382 raft_consensus.cc:504] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.654294 7382 leader_election.cc:288] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:59.654596 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ecdda0ff32d4e0fac0b94588bb9d9e1" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:59.654628 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ecdda0ff32d4e0fac0b94588bb9d9e1" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:59.654747 30406 raft_consensus.cc:2454] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:59.654757 32416 raft_consensus.cc:2454] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 0.
I20240501 14:09:59.655110 32752 leader_election.cc:302] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:09:59.655295 7382 raft_consensus.cc:2781] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:59.655354 7382 raft_consensus.cc:482] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:59.655401 7382 raft_consensus.cc:3037] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.656412 7382 raft_consensus.cc:504] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.656723 7382 leader_election.cc:288] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:59.657003 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ecdda0ff32d4e0fac0b94588bb9d9e1" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:09:59.657116 30406 raft_consensus.cc:3037] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.657120 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ecdda0ff32d4e0fac0b94588bb9d9e1" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:59.657248 32416 raft_consensus.cc:3037] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.658077 30406 raft_consensus.cc:2454] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:59.658135 32416 raft_consensus.cc:2454] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:09:59.658407 32750 leader_election.cc:302] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:59.658618 7382 raft_consensus.cc:2781] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:59.658687 7382 raft_consensus.cc:686] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:09:59.658829 7382 consensus_queue.cc:237] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.661475 29866 catalog_manager.cc:5484] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:59.735081 6185 consensus_queue.cc:1035] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:59.737850 29872 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:59.740446 8442 consensus_queue.cc:1035] T 60090b33d0e24ff18fd6c73548b05a13 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:59.756616 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 77b0cc955c704a449d1b0333a678d3b8 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=e27193193d284b68924b3dc3a8acbbd5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:59.756814 327 tablet_service.cc:1467] Processing CreateTablet for tablet 77b0cc955c704a449d1b0333a678d3b8 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=e27193193d284b68924b3dc3a8acbbd5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:59.756906 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77b0cc955c704a449d1b0333a678d3b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.757058 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77b0cc955c704a449d1b0333a678d3b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.759246 9521 tablet_bootstrap.cc:492] T 77b0cc955c704a449d1b0333a678d3b8 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:09:59.759516 30126 tablet_service.cc:1467] Processing CreateTablet for tablet d69d3c347a6d41b69adc66417a9bb8a4 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=e27193193d284b68924b3dc3a8acbbd5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:59.760545 9521 tablet_bootstrap.cc:654] T 77b0cc955c704a449d1b0333a678d3b8 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.763384 32664 tablet_service.cc:1467] Processing CreateTablet for tablet a3151db71f204f74a4c958093a1b1b06 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=e27193193d284b68924b3dc3a8acbbd5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:59.763610 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3151db71f204f74a4c958093a1b1b06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.763819 9521 tablet_bootstrap.cc:492] T 77b0cc955c704a449d1b0333a678d3b8 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:09:59.763902 9521 ts_tablet_manager.cc:1367] T 77b0cc955c704a449d1b0333a678d3b8 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:59.759766 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d69d3c347a6d41b69adc66417a9bb8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.766187 9605 tablet_bootstrap.cc:492] T d69d3c347a6d41b69adc66417a9bb8a4 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:59.761749 30125 tablet_service.cc:1467] Processing CreateTablet for tablet a3151db71f204f74a4c958093a1b1b06 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=e27193193d284b68924b3dc3a8acbbd5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:59.766425 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3151db71f204f74a4c958093a1b1b06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.767292 9605 tablet_bootstrap.cc:654] T d69d3c347a6d41b69adc66417a9bb8a4 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.768323 9605 tablet_bootstrap.cc:492] T d69d3c347a6d41b69adc66417a9bb8a4 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:59.768391 9605 ts_tablet_manager.cc:1367] T d69d3c347a6d41b69adc66417a9bb8a4 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:59.768834 9605 raft_consensus.cc:348] T d69d3c347a6d41b69adc66417a9bb8a4 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.768945 9605 raft_consensus.cc:374] T d69d3c347a6d41b69adc66417a9bb8a4 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.768978 9605 raft_consensus.cc:729] T d69d3c347a6d41b69adc66417a9bb8a4 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.769119 9605 consensus_queue.cc:260] T d69d3c347a6d41b69adc66417a9bb8a4 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.769413 9605 ts_tablet_manager.cc:1398] T d69d3c347a6d41b69adc66417a9bb8a4 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:59.769554 9605 tablet_bootstrap.cc:492] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:09:59.769572 9521 raft_consensus.cc:348] T 77b0cc955c704a449d1b0333a678d3b8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:59.769663 9521 raft_consensus.cc:374] T 77b0cc955c704a449d1b0333a678d3b8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.769690 9521 raft_consensus.cc:729] T 77b0cc955c704a449d1b0333a678d3b8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.769778 9521 consensus_queue.cc:260] T 77b0cc955c704a449d1b0333a678d3b8 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:59.770107 30388 tablet_service.cc:1467] Processing CreateTablet for tablet d69d3c347a6d41b69adc66417a9bb8a4 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=e27193193d284b68924b3dc3a8acbbd5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:59.770381 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d69d3c347a6d41b69adc66417a9bb8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.770404 9605 tablet_bootstrap.cc:654] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.773378 9521 ts_tablet_manager.cc:1398] T 77b0cc955c704a449d1b0333a678d3b8 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240501 14:09:59.773797 9519 tablet_bootstrap.cc:492] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:09:59.774752 9519 tablet_bootstrap.cc:654] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.778157 9603 tablet_bootstrap.cc:492] T 77b0cc955c704a449d1b0333a678d3b8 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:59.778625 30257 tablet_service.cc:1467] Processing CreateTablet for tablet a3151db71f204f74a4c958093a1b1b06 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=e27193193d284b68924b3dc3a8acbbd5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:59.778864 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3151db71f204f74a4c958093a1b1b06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.779112 32400 tablet_service.cc:1467] Processing CreateTablet for tablet d69d3c347a6d41b69adc66417a9bb8a4 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=e27193193d284b68924b3dc3a8acbbd5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:59.779119 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 77b0cc955c704a449d1b0333a678d3b8 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=e27193193d284b68924b3dc3a8acbbd5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:59.779352 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d69d3c347a6d41b69adc66417a9bb8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.779363 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77b0cc955c704a449d1b0333a678d3b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.783133 29864 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:59.783723 9518 tablet_bootstrap.cc:492] T d69d3c347a6d41b69adc66417a9bb8a4 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:09:59.783825 9609 tablet_bootstrap.cc:492] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:09:59.785367 29993 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:59.785652 32533 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:09:59.786720 9518 tablet_bootstrap.cc:654] T d69d3c347a6d41b69adc66417a9bb8a4 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.788715 9609 tablet_bootstrap.cc:654] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.789909 9609 tablet_bootstrap.cc:492] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:09:59.789983 9609 ts_tablet_manager.cc:1367] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:59.790400 9609 raft_consensus.cc:348] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:59.790509 9609 raft_consensus.cc:374] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.790544 9609 raft_consensus.cc:729] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.790652 9609 consensus_queue.cc:260] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:59.790867 30406 raft_consensus.cc:1261] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:59.790926 9609 ts_tablet_manager.cc:1398] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:59.791425 32416 raft_consensus.cc:1261] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:59.791429 7382 consensus_queue.cc:1035] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:59.792070 7382 consensus_queue.cc:1035] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:59.796063 9416 tablet.cc:1722] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:59.796341 9416 tablet.cc:1620] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:59.796500 9415 tablet.cc:1722] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:59.796753 9415 tablet.cc:1620] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:59.799312 29864 catalog_manager.cc:5576] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265) does not have the latest schema for tablet 8ecdda0ff32d4e0fac0b94588bb9d9e1 (table rebalancer_dynamic_table [id=51ac4aecbc2747528a1f54e5c7977ba7]). Expected version 1 got 0
I20240501 14:09:59.804087 29872 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=51ac4aecbc2747528a1f54e5c7977ba7] alter complete (version 1)
I20240501 14:09:59.809710 9605 tablet_bootstrap.cc:492] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:09:59.809824 9605 ts_tablet_manager.cc:1367] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.000s
I20240501 14:09:59.810305 9605 raft_consensus.cc:348] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:59.810415 9605 raft_consensus.cc:374] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.810447 9605 raft_consensus.cc:729] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.810549 9605 consensus_queue.cc:260] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:59.810758 7205 raft_consensus.cc:482] T d69d3c347a6d41b69adc66417a9bb8a4 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:59.810828 7205 raft_consensus.cc:504] T d69d3c347a6d41b69adc66417a9bb8a4 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.811177 7205 leader_election.cc:288] T d69d3c347a6d41b69adc66417a9bb8a4 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:59.811612 30257 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:59.812917 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69d3c347a6d41b69adc66417a9bb8a4" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:59.812975 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69d3c347a6d41b69adc66417a9bb8a4" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:09:59.813377 29867 catalog_manager.cc:5576] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357) does not have the latest schema for tablet 8ecdda0ff32d4e0fac0b94588bb9d9e1 (table rebalancer_dynamic_table [id=51ac4aecbc2747528a1f54e5c7977ba7]). Expected version 1 got 0
W20240501 14:09:59.813419 30079 leader_election.cc:341] T d69d3c347a6d41b69adc66417a9bb8a4 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:59.813822 30388 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:09:59.813853 30081 leader_election.cc:341] T d69d3c347a6d41b69adc66417a9bb8a4 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:59.813921 30081 leader_election.cc:302] T d69d3c347a6d41b69adc66417a9bb8a4 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d35c16dcb506426993811bb2519d7db3; no voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7
I20240501 14:09:59.814148 9529 raft_consensus.cc:2726] T d69d3c347a6d41b69adc66417a9bb8a4 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:09:59.810770 9605 ts_tablet_manager.cc:1398] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:59.815208 32400 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:09:59.820667 9603 tablet_bootstrap.cc:654] T 77b0cc955c704a449d1b0333a678d3b8 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.825309 9610 tablet_bootstrap.cc:492] T a3151db71f204f74a4c958093a1b1b06 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:09:59.827173 9610 tablet_bootstrap.cc:654] T a3151db71f204f74a4c958093a1b1b06 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.831374 29867 catalog_manager.cc:5576] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357) does not have the latest schema for tablet 8ecdda0ff32d4e0fac0b94588bb9d9e1 (table rebalancer_dynamic_table [id=51ac4aecbc2747528a1f54e5c7977ba7]). Expected version 1 got 0
I20240501 14:09:59.835062 9518 tablet_bootstrap.cc:492] T d69d3c347a6d41b69adc66417a9bb8a4 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:09:59.835158 9518 ts_tablet_manager.cc:1367] T d69d3c347a6d41b69adc66417a9bb8a4 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.051s user 0.000s sys 0.002s
I20240501 14:09:59.835644 9518 raft_consensus.cc:348] T d69d3c347a6d41b69adc66417a9bb8a4 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.835752 9518 raft_consensus.cc:374] T d69d3c347a6d41b69adc66417a9bb8a4 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.835778 9518 raft_consensus.cc:729] T d69d3c347a6d41b69adc66417a9bb8a4 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.835891 9518 consensus_queue.cc:260] T d69d3c347a6d41b69adc66417a9bb8a4 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.836158 9518 ts_tablet_manager.cc:1398] T d69d3c347a6d41b69adc66417a9bb8a4 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:59.841900 327 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:09:59.842717 9603 tablet_bootstrap.cc:492] T 77b0cc955c704a449d1b0333a678d3b8 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:59.842847 9603 ts_tablet_manager.cc:1367] T 77b0cc955c704a449d1b0333a678d3b8 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.000s
I20240501 14:09:59.843529 9603 raft_consensus.cc:348] T 77b0cc955c704a449d1b0333a678d3b8 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:59.843679 9603 raft_consensus.cc:374] T 77b0cc955c704a449d1b0333a678d3b8 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.843739 9603 raft_consensus.cc:729] T 77b0cc955c704a449d1b0333a678d3b8 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.843887 9603 consensus_queue.cc:260] T 77b0cc955c704a449d1b0333a678d3b8 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:09:59.844293 9603 ts_tablet_manager.cc:1398] T 77b0cc955c704a449d1b0333a678d3b8 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:59.845204 9603 tablet_bootstrap.cc:492] T a3151db71f204f74a4c958093a1b1b06 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:09:59.846429 9603 tablet_bootstrap.cc:654] T a3151db71f204f74a4c958093a1b1b06 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.851493 9603 tablet_bootstrap.cc:492] T a3151db71f204f74a4c958093a1b1b06 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:09:59.851617 9603 ts_tablet_manager.cc:1367] T a3151db71f204f74a4c958093a1b1b06 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:59.852097 9603 raft_consensus.cc:348] T a3151db71f204f74a4c958093a1b1b06 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:59.852308 9603 raft_consensus.cc:374] T a3151db71f204f74a4c958093a1b1b06 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.852377 9603 raft_consensus.cc:729] T a3151db71f204f74a4c958093a1b1b06 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.852547 9603 consensus_queue.cc:260] T a3151db71f204f74a4c958093a1b1b06 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:59.854805 9519 tablet_bootstrap.cc:492] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:09:59.854907 9519 ts_tablet_manager.cc:1367] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.081s user 0.002s sys 0.000s
I20240501 14:09:59.855368 9603 ts_tablet_manager.cc:1398] T a3151db71f204f74a4c958093a1b1b06 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:09:59.855459 9519 raft_consensus.cc:348] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.855558 9519 raft_consensus.cc:374] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.855589 9519 raft_consensus.cc:729] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.855698 9519 consensus_queue.cc:260] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.855963 9519 ts_tablet_manager.cc:1398] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:59.860102 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:09:59.860337 9610 tablet_bootstrap.cc:492] T a3151db71f204f74a4c958093a1b1b06 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:09:59.860455 9610 ts_tablet_manager.cc:1367] T a3151db71f204f74a4c958093a1b1b06 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20240501 14:09:59.861047 9610 raft_consensus.cc:348] T a3151db71f204f74a4c958093a1b1b06 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:59.861198 9610 raft_consensus.cc:374] T a3151db71f204f74a4c958093a1b1b06 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.861268 9610 raft_consensus.cc:729] T a3151db71f204f74a4c958093a1b1b06 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.861429 9610 consensus_queue.cc:260] T a3151db71f204f74a4c958093a1b1b06 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:59.862577 9610 ts_tablet_manager.cc:1398] T a3151db71f204f74a4c958093a1b1b06 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:59.878975 32664 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:59.893637 9616 tablet.cc:1722] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:59.894147 9616 tablet.cc:1620] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7: MemRowSet was empty: no flush needed.
I20240501 14:09:59.898094 8442 raft_consensus.cc:482] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:59.898185 8442 raft_consensus.cc:504] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.898538 8442 leader_election.cc:288] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:59.899055 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69d3c347a6d41b69adc66417a9bb8a4" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:09:59.899171 30145 raft_consensus.cc:2454] T d69d3c347a6d41b69adc66417a9bb8a4 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:09:59.899646 30344 leader_election.cc:302] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:59.899814 6185 raft_consensus.cc:2781] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:59.899865 6185 raft_consensus.cc:482] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:59.899905 6185 raft_consensus.cc:3037] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.901062 6185 raft_consensus.cc:504] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.901398 6185 leader_election.cc:288] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:09:59.901872 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69d3c347a6d41b69adc66417a9bb8a4" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:09:59.901994 32415 raft_consensus.cc:3037] T d69d3c347a6d41b69adc66417a9bb8a4 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.902791 32415 raft_consensus.cc:2454] T d69d3c347a6d41b69adc66417a9bb8a4 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:59.902951 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69d3c347a6d41b69adc66417a9bb8a4" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:09:59.903051 32416 raft_consensus.cc:2362] T d69d3c347a6d41b69adc66417a9bb8a4 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1. Re-sending same reply.
I20240501 14:09:59.903040 30343 leader_election.cc:302] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:09:59.903201 9619 raft_consensus.cc:2781] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:59.903379 9619 raft_consensus.cc:686] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:09:59.903436 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69d3c347a6d41b69adc66417a9bb8a4" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:09:59.903542 30146 raft_consensus.cc:3037] T d69d3c347a6d41b69adc66417a9bb8a4 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.903532 9619 consensus_queue.cc:237] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:09:59.904521 30146 raft_consensus.cc:2454] T d69d3c347a6d41b69adc66417a9bb8a4 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:09:59.905864 29864 catalog_manager.cc:5484] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:09:59.913201 29864 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:59.919271 9416 tablet.cc:1722] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:59.919570 9416 tablet.cc:1620] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849: MemRowSet was empty: no flush needed.
I20240501 14:09:59.920437 9415 tablet.cc:1722] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:59.920755 9415 tablet.cc:1620] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3: MemRowSet was empty: no flush needed.
I20240501 14:09:59.924062 9616 tablet.cc:1722] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:59.924384 9616 tablet.cc:1620] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7: MemRowSet was empty: no flush needed.
I20240501 14:09:59.930639 29864 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=51ac4aecbc2747528a1f54e5c7977ba7] alter complete (version 2)
I20240501 14:09:59.933024 30146 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5518112449d9448e98f5c779f1fa3248"
dest_uuid: "d35c16dcb506426993811bb2519d7db3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53452
I20240501 14:09:59.933148 30146 raft_consensus.cc:595] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:59.995045 7205 raft_consensus.cc:482] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:59.995158 7205 raft_consensus.cc:504] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:59.995527 7205 leader_election.cc:288] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:09:59.996340 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3151db71f204f74a4c958093a1b1b06" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:09:59.996351 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3151db71f204f74a4c958093a1b1b06" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:09:59.996469 30277 raft_consensus.cc:2454] T a3151db71f204f74a4c958093a1b1b06 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:59.996469 32684 raft_consensus.cc:2454] T a3151db71f204f74a4c958093a1b1b06 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 0.
I20240501 14:09:59.997009 30081 leader_election.cc:302] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:09:59.997373 7205 raft_consensus.cc:2781] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:59.997434 7205 raft_consensus.cc:482] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:59.997470 7205 raft_consensus.cc:3037] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.998576 7205 raft_consensus.cc:504] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:09:59.998919 7205 leader_election.cc:288] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:00.000108 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3151db71f204f74a4c958093a1b1b06" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:10:00.000255 32684 raft_consensus.cc:3037] T a3151db71f204f74a4c958093a1b1b06 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.000522 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3151db71f204f74a4c958093a1b1b06" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:10:00.000635 30277 raft_consensus.cc:3037] T a3151db71f204f74a4c958093a1b1b06 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.001338 32684 raft_consensus.cc:2454] T a3151db71f204f74a4c958093a1b1b06 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:10:00.001713 30277 raft_consensus.cc:2454] T a3151db71f204f74a4c958093a1b1b06 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:10:00.003562 30079 leader_election.cc:302] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d35c16dcb506426993811bb2519d7db3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:10:00.006114 7205 raft_consensus.cc:2781] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:00.006193 7205 raft_consensus.cc:686] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:10:00.006314 7205 consensus_queue.cc:237] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:00.008594 29865 catalog_manager.cc:5484] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:10:00.026939 9452 raft_consensus.cc:482] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:00.027056 9452 raft_consensus.cc:504] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:00.027439 9452 leader_election.cc:288] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:10:00.027839 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b0cc955c704a449d1b0333a678d3b8" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:10:00.027974 32684 raft_consensus.cc:2454] T 77b0cc955c704a449d1b0333a678d3b8 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:10:00.028304 29948 leader_election.cc:302] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:10:00.028565 9452 raft_consensus.cc:2781] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:00.028625 9452 raft_consensus.cc:482] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:00.028666 9452 raft_consensus.cc:3037] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.029692 9452 raft_consensus.cc:504] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:00.030012 9452 leader_election.cc:288] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:10:00.030179 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b0cc955c704a449d1b0333a678d3b8" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:10:00.030306 348 raft_consensus.cc:2454] T 77b0cc955c704a449d1b0333a678d3b8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:10:00.030943 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b0cc955c704a449d1b0333a678d3b8" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:10:00.031042 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b0cc955c704a449d1b0333a678d3b8" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:10:00.031072 348 raft_consensus.cc:3037] T 77b0cc955c704a449d1b0333a678d3b8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.031132 32684 raft_consensus.cc:3037] T 77b0cc955c704a449d1b0333a678d3b8 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.032007 348 raft_consensus.cc:2454] T 77b0cc955c704a449d1b0333a678d3b8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:10:00.032002 32684 raft_consensus.cc:2454] T 77b0cc955c704a449d1b0333a678d3b8 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:10:00.032809 29948 leader_election.cc:302] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:10:00.036425 9452 raft_consensus.cc:2781] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:00.036531 9452 raft_consensus.cc:686] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:10:00.036653 9452 consensus_queue.cc:237] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:00.039245 29869 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:10:00.040299 29872 catalog_manager.cc:5484] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:00.046452 9416 tablet.cc:1722] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:10:00.047047 9616 tablet.cc:1722] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:10:00.054606 29865 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=51ac4aecbc2747528a1f54e5c7977ba7] alter complete (version 3)
I20240501 14:10:00.056418 9415 tablet.cc:1722] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:10:00.081996 32533 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:00.090394 29993 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:00.090958 30388 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:00.096807 32664 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:10:00.111666 30125 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:10:00.113338 7205 raft_consensus.cc:979] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3: : Instructing follower a91029b90bf9415a89a2f70736759ad3 to start an election
I20240501 14:10:00.113417 7205 raft_consensus.cc:1067] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Signalling peer a91029b90bf9415a89a2f70736759ad3 to start an election
I20240501 14:10:00.114005 32416 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5518112449d9448e98f5c779f1fa3248"
dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
from {username='slave'} at 127.22.252.130:47505
I20240501 14:10:00.114166 32416 raft_consensus.cc:482] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:00.114228 32416 raft_consensus.cc:3037] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:00.115353 32416 raft_consensus.cc:504] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:00.115736 32416 leader_election.cc:288] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 election: Requested vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:10:00.116744 327 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:10:00.117194 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5518112449d9448e98f5c779f1fa3248" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:10:00.117311 30264 raft_consensus.cc:3037] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:00.118278 30264 raft_consensus.cc:2454] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 2.
I20240501 14:10:00.120568 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5518112449d9448e98f5c779f1fa3248" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:10:00.120689 32554 raft_consensus.cc:3037] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:00.121577 32554 raft_consensus.cc:2454] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 2.
I20240501 14:10:00.123724 32359 leader_election.cc:302] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 5f8a0cfdfdca4148a75686dc88059cf3, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:10:00.124276 30257 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:00.124320 7305 raft_consensus.cc:2781] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:00.124408 7305 raft_consensus.cc:686] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 [term 2 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:10:00.124562 7305 consensus_queue.cc:237] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:00.124737 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5518112449d9448e98f5c779f1fa3248" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:10:00.124850 30146 raft_consensus.cc:3032] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:10:00.124889 30146 raft_consensus.cc:729] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:10:00.125023 30146 consensus_queue.cc:260] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:00.125242 30146 raft_consensus.cc:3037] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:00.126477 30146 raft_consensus.cc:2454] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 2.
I20240501 14:10:00.127563 29872 catalog_manager.cc:5484] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 1 to 2, leader changed from d35c16dcb506426993811bb2519d7db3 (127.22.252.130) to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 2 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:00.144423 32400 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:10:00.157868 29872 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:10:00.158035 29872 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:10:00.160696 29872 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 8ecdda0ff32d4e0fac0b94588bb9d9e1
I20240501 14:10:00.161307 32400 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ecdda0ff32d4e0fac0b94588bb9d9e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:10:00 UTC) from {username='slave'} at 127.0.0.1:53090
I20240501 14:10:00.161423 327 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ecdda0ff32d4e0fac0b94588bb9d9e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:10:00 UTC) from {username='slave'} at 127.0.0.1:45868
I20240501 14:10:00.161562 30388 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ecdda0ff32d4e0fac0b94588bb9d9e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:10:00 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:10:00.161962 9682 tablet_replica.cc:330] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:10:00.162154 9682 raft_consensus.cc:2229] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:00.162329 9682 raft_consensus.cc:2258] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:00.162938 9682 ts_tablet_manager.cc:1875] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:10:00.165525 9681 tablet_replica.cc:330] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:10:00.165763 9681 raft_consensus.cc:2229] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:10:00.166042 9681 raft_consensus.cc:2258] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:00.166090 9682 ts_tablet_manager.cc:1888] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:10:00.166254 9682 log.cc:1196] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/8ecdda0ff32d4e0fac0b94588bb9d9e1
I20240501 14:10:00.166673 9682 ts_tablet_manager.cc:1909] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:10:00.166740 9681 ts_tablet_manager.cc:1875] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:10:00.167501 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 8ecdda0ff32d4e0fac0b94588bb9d9e1 (table rebalancer_dynamic_table_new_name [id=51ac4aecbc2747528a1f54e5c7977ba7]) successfully deleted
I20240501 14:10:00.169148 9681 ts_tablet_manager.cc:1888] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:10:00.169224 9681 log.cc:1196] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/8ecdda0ff32d4e0fac0b94588bb9d9e1
I20240501 14:10:00.169476 9681 ts_tablet_manager.cc:1909] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting consensus metadata
I20240501 14:10:00.170070 9680 tablet_replica.cc:330] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:10:00.170064 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 8ecdda0ff32d4e0fac0b94588bb9d9e1 (table rebalancer_dynamic_table_new_name [id=51ac4aecbc2747528a1f54e5c7977ba7]) successfully deleted
I20240501 14:10:00.170220 9680 raft_consensus.cc:2229] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:00.170333 9680 raft_consensus.cc:2258] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:00.170854 9680 ts_tablet_manager.cc:1875] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:10:00.173005 9680 ts_tablet_manager.cc:1888] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:10:00.173141 9680 log.cc:1196] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/8ecdda0ff32d4e0fac0b94588bb9d9e1
I20240501 14:10:00.173403 9680 ts_tablet_manager.cc:1909] T 8ecdda0ff32d4e0fac0b94588bb9d9e1 P a91029b90bf9415a89a2f70736759ad3: Deleting consensus metadata
I20240501 14:10:00.173954 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 8ecdda0ff32d4e0fac0b94588bb9d9e1 (table rebalancer_dynamic_table_new_name [id=51ac4aecbc2747528a1f54e5c7977ba7]) successfully deleted
I20240501 14:10:00.262678 29868 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:10:00.269614 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 36e60603bc30424fb583de6cc7db8207 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a9126bd559ea4e80ae42b4c9ad45b64d]), partition=
I20240501 14:10:00.269888 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36e60603bc30424fb583de6cc7db8207. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.272363 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 36e60603bc30424fb583de6cc7db8207 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a9126bd559ea4e80ae42b4c9ad45b64d]), partition=
I20240501 14:10:00.272612 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36e60603bc30424fb583de6cc7db8207. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.274721 9609 tablet_bootstrap.cc:492] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:10:00.275698 9609 tablet_bootstrap.cc:654] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.276386 9519 tablet_bootstrap.cc:492] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:10:00.277254 9519 tablet_bootstrap.cc:654] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.278914 30125 tablet_service.cc:1467] Processing CreateTablet for tablet 36e60603bc30424fb583de6cc7db8207 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a9126bd559ea4e80ae42b4c9ad45b64d]), partition=
I20240501 14:10:00.279158 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36e60603bc30424fb583de6cc7db8207. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.281458 9605 tablet_bootstrap.cc:492] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:10:00.282266 9605 tablet_bootstrap.cc:654] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.282601 9519 tablet_bootstrap.cc:492] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:10:00.282663 9519 ts_tablet_manager.cc:1367] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:10:00.283102 9519 raft_consensus.cc:348] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:00.283205 9519 raft_consensus.cc:374] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.283267 9519 raft_consensus.cc:729] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.283385 9519 consensus_queue.cc:260] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:00.283720 9519 ts_tablet_manager.cc:1398] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:00.283903 9605 tablet_bootstrap.cc:492] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:10:00.284009 9605 ts_tablet_manager.cc:1367] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:00.285569 9605 raft_consensus.cc:348] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:00.285676 9605 raft_consensus.cc:374] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.285704 9605 raft_consensus.cc:729] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.285806 9605 consensus_queue.cc:260] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:00.286121 9605 ts_tablet_manager.cc:1398] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:10:00.297583 9609 tablet_bootstrap.cc:492] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:10:00.297683 9609 ts_tablet_manager.cc:1367] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.001s
I20240501 14:10:00.298125 9609 raft_consensus.cc:348] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:00.298224 9609 raft_consensus.cc:374] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.298254 9609 raft_consensus.cc:729] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.298389 9609 consensus_queue.cc:260] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:00.298704 9609 ts_tablet_manager.cc:1398] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:00.348640 32533 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:00.352209 30257 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:00.352937 29993 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:00.353405 32400 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:10:00.355841 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:00.356532 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:00.356714 30388 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:00.366919 9620 consensus_queue.cc:1035] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.371742 9619 consensus_queue.cc:1035] T d69d3c347a6d41b69adc66417a9bb8a4 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.373472 32664 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:10:00.456023 30406 consensus_queue.cc:237] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:00.457340 30277 raft_consensus.cc:1261] T b0ef35582c4f4d5c95199e00ccac92c9 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:00.457648 32416 raft_consensus.cc:1261] T b0ef35582c4f4d5c95199e00ccac92c9 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:00.457695 9619 consensus_queue.cc:1035] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:00.458014 9620 consensus_queue.cc:1035] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:00.459592 9619 raft_consensus.cc:2932] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:10:00.460096 30277 raft_consensus.cc:2932] T b0ef35582c4f4d5c95199e00ccac92c9 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:10:00.461668 32416 raft_consensus.cc:2932] T b0ef35582c4f4d5c95199e00ccac92c9 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:10:00.463791 29868 catalog_manager.cc:5484] T b0ef35582c4f4d5c95199e00ccac92c9 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:10:00.467622 32533 tablet_service.cc:1514] Processing DeleteTablet for tablet b0ef35582c4f4d5c95199e00ccac92c9 with delete_type TABLET_DATA_TOMBSTONED (TS 5f8a0cfdfdca4148a75686dc88059cf3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47638
I20240501 14:10:00.468022 7378 consensus_queue.cc:1035] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.468448 9766 tablet_replica.cc:330] T b0ef35582c4f4d5c95199e00ccac92c9 P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:10:00.468614 9766 raft_consensus.cc:2229] T b0ef35582c4f4d5c95199e00ccac92c9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:00.468796 9766 raft_consensus.cc:2258] T b0ef35582c4f4d5c95199e00ccac92c9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:00.469302 9766 ts_tablet_manager.cc:1875] T b0ef35582c4f4d5c95199e00ccac92c9 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:00.471518 9765 consensus_queue.cc:1035] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.471872 9766 ts_tablet_manager.cc:1888] T b0ef35582c4f4d5c95199e00ccac92c9 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:10:00.472080 9766 log.cc:1196] T b0ef35582c4f4d5c95199e00ccac92c9 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/b0ef35582c4f4d5c95199e00ccac92c9
I20240501 14:10:00.472846 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet b0ef35582c4f4d5c95199e00ccac92c9 (table rebalancer_extra_table_16 [id=6139b5f2fcc94630ac7d807b61ccd1ff]) successfully deleted
I20240501 14:10:00.489881 9529 consensus_queue.cc:1035] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.493752 9529 consensus_queue.cc:1035] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.546481 32554 raft_consensus.cc:1261] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 2 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:10:00.546883 7305 consensus_queue.cc:1035] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:00.570541 29866 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:10:00.577312 32400 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5188c15666488c99717f68acfeeea4 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=1c25ee4d00414b758f0edeb63b4b6ba4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.577631 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5188c15666488c99717f68acfeeea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.578104 327 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca6bdb2e17a4e5e9dab845cad66ada8 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=1c25ee4d00414b758f0edeb63b4b6ba4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.578163 32401 tablet_service.cc:1467] Processing CreateTablet for tablet 0f980c414f0244b19203b18a0a3f1978 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=1c25ee4d00414b758f0edeb63b4b6ba4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.578398 32401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f980c414f0244b19203b18a0a3f1978. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.578393 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca6bdb2e17a4e5e9dab845cad66ada8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.580617 30257 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5188c15666488c99717f68acfeeea4 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=1c25ee4d00414b758f0edeb63b4b6ba4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.580899 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5188c15666488c99717f68acfeeea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.581036 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca6bdb2e17a4e5e9dab845cad66ada8 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=1c25ee4d00414b758f0edeb63b4b6ba4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.581288 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca6bdb2e17a4e5e9dab845cad66ada8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.582273 9818 tablet_bootstrap.cc:492] T 9e5188c15666488c99717f68acfeeea4 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:10:00.583415 9818 tablet_bootstrap.cc:654] T 9e5188c15666488c99717f68acfeeea4 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.583415 30255 tablet_service.cc:1467] Processing CreateTablet for tablet 0f980c414f0244b19203b18a0a3f1978 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=1c25ee4d00414b758f0edeb63b4b6ba4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.583650 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f980c414f0244b19203b18a0a3f1978. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.583698 329 tablet_service.cc:1467] Processing CreateTablet for tablet 0f980c414f0244b19203b18a0a3f1978 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=1c25ee4d00414b758f0edeb63b4b6ba4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.583909 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f980c414f0244b19203b18a0a3f1978. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.584089 9820 tablet_bootstrap.cc:492] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:10:00.585875 9820 tablet_bootstrap.cc:654] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.585868 32533 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5188c15666488c99717f68acfeeea4 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=1c25ee4d00414b758f0edeb63b4b6ba4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.586148 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5188c15666488c99717f68acfeeea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.586222 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca6bdb2e17a4e5e9dab845cad66ada8 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=1c25ee4d00414b758f0edeb63b4b6ba4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.586493 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca6bdb2e17a4e5e9dab845cad66ada8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.590377 30145 raft_consensus.cc:1261] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 2 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:10:00.591478 9729 consensus_queue.cc:1035] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:00.594033 9818 tablet_bootstrap.cc:492] T 9e5188c15666488c99717f68acfeeea4 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:10:00.594115 9818 ts_tablet_manager.cc:1367] T 9e5188c15666488c99717f68acfeeea4 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:10:00.596509 9818 raft_consensus.cc:348] T 9e5188c15666488c99717f68acfeeea4 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:00.596684 9818 raft_consensus.cc:374] T 9e5188c15666488c99717f68acfeeea4 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.596755 9818 raft_consensus.cc:729] T 9e5188c15666488c99717f68acfeeea4 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.596958 9818 consensus_queue.cc:260] T 9e5188c15666488c99717f68acfeeea4 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:00.598520 9818 ts_tablet_manager.cc:1398] T 9e5188c15666488c99717f68acfeeea4 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:10:00.598721 9818 tablet_bootstrap.cc:492] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:10:00.598824 30264 raft_consensus.cc:1261] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:10:00.599182 9729 consensus_queue.cc:1035] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:00.604188 9819 tablet_bootstrap.cc:492] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:10:00.605630 9818 tablet_bootstrap.cc:654] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.598178 9822 tablet_bootstrap.cc:492] T 0f980c414f0244b19203b18a0a3f1978 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:10:00.606478 9822 tablet_bootstrap.cc:654] T 0f980c414f0244b19203b18a0a3f1978 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.607335 9824 tablet_bootstrap.cc:492] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:10:00.607856 9820 tablet_bootstrap.cc:492] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:10:00.607993 9820 ts_tablet_manager.cc:1367] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240501 14:10:00.608561 9819 tablet_bootstrap.cc:654] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.608743 9820 raft_consensus.cc:348] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:00.608899 9820 raft_consensus.cc:374] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.608969 9820 raft_consensus.cc:729] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.609171 9820 consensus_queue.cc:260] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:00.609509 9820 ts_tablet_manager.cc:1398] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:00.609740 9820 tablet_bootstrap.cc:492] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:10:00.610352 9824 tablet_bootstrap.cc:654] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.612655 9819 tablet_bootstrap.cc:492] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:10:00.612803 9819 ts_tablet_manager.cc:1367] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20240501 14:10:00.613314 9819 raft_consensus.cc:348] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:00.613492 9819 raft_consensus.cc:374] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.613569 9819 raft_consensus.cc:729] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.613734 9819 consensus_queue.cc:260] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:00.617795 9820 tablet_bootstrap.cc:654] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.619505 9819 ts_tablet_manager.cc:1398] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240501 14:10:00.620375 9818 tablet_bootstrap.cc:492] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3: No bootstrap required, opened a new log
I20240501 14:10:00.620642 9818 ts_tablet_manager.cc:1367] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240501 14:10:00.621560 9819 tablet_bootstrap.cc:492] T 0f980c414f0244b19203b18a0a3f1978 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:10:00.621181 9818 raft_consensus.cc:348] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:00.622579 9818 raft_consensus.cc:374] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.622648 9818 raft_consensus.cc:729] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.622815 9818 consensus_queue.cc:260] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:00.623625 9819 tablet_bootstrap.cc:654] T 0f980c414f0244b19203b18a0a3f1978 P 1d0bb69611af4876a2b3b33c4766e13a: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.624902 9819 tablet_bootstrap.cc:492] T 0f980c414f0244b19203b18a0a3f1978 P 1d0bb69611af4876a2b3b33c4766e13a: No bootstrap required, opened a new log
I20240501 14:10:00.624994 9819 ts_tablet_manager.cc:1367] T 0f980c414f0244b19203b18a0a3f1978 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:00.625438 9819 raft_consensus.cc:348] T 0f980c414f0244b19203b18a0a3f1978 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:00.625526 9819 raft_consensus.cc:374] T 0f980c414f0244b19203b18a0a3f1978 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.625552 9819 raft_consensus.cc:729] T 0f980c414f0244b19203b18a0a3f1978 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.625665 9819 consensus_queue.cc:260] T 0f980c414f0244b19203b18a0a3f1978 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:00.625692 9818 ts_tablet_manager.cc:1398] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240501 14:10:00.625913 9819 ts_tablet_manager.cc:1398] T 0f980c414f0244b19203b18a0a3f1978 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:00.630995 9820 tablet_bootstrap.cc:492] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:10:00.632180 9820 ts_tablet_manager.cc:1367] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240501 14:10:00.632652 9820 raft_consensus.cc:348] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:00.632763 9820 raft_consensus.cc:374] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.632848 9820 raft_consensus.cc:729] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.632966 9820 consensus_queue.cc:260] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:00.633316 9820 ts_tablet_manager.cc:1398] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:00.637967 9822 tablet_bootstrap.cc:492] T 0f980c414f0244b19203b18a0a3f1978 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:10:00.638092 9822 ts_tablet_manager.cc:1367] T 0f980c414f0244b19203b18a0a3f1978 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.000s
I20240501 14:10:00.638661 9822 raft_consensus.cc:348] T 0f980c414f0244b19203b18a0a3f1978 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:00.638780 9822 raft_consensus.cc:374] T 0f980c414f0244b19203b18a0a3f1978 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.638815 9822 raft_consensus.cc:729] T 0f980c414f0244b19203b18a0a3f1978 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.638927 9822 consensus_queue.cc:260] T 0f980c414f0244b19203b18a0a3f1978 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:00.639204 9822 ts_tablet_manager.cc:1398] T 0f980c414f0244b19203b18a0a3f1978 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:10:00.639410 9822 tablet_bootstrap.cc:492] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:10:00.641281 9822 tablet_bootstrap.cc:654] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.642444 9822 tablet_bootstrap.cc:492] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:10:00.642530 9822 ts_tablet_manager.cc:1367] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:10:00.643253 9824 tablet_bootstrap.cc:492] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:10:00.643355 9824 ts_tablet_manager.cc:1367] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20240501 14:10:00.643862 9824 raft_consensus.cc:348] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:00.643950 9824 raft_consensus.cc:374] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.643978 9824 raft_consensus.cc:729] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.644088 9824 consensus_queue.cc:260] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:00.644626 9824 ts_tablet_manager.cc:1398] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:00.644979 9822 raft_consensus.cc:348] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:00.645067 9822 raft_consensus.cc:374] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.645108 9822 raft_consensus.cc:729] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.645247 9822 consensus_queue.cc:260] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:00.646382 9822 ts_tablet_manager.cc:1398] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:10:00.657430 9412 raft_consensus.cc:482] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:00.657532 9412 raft_consensus.cc:504] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:00.657881 9412 leader_election.cc:288] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:00.658329 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5188c15666488c99717f68acfeeea4" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:10:00.658409 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5188c15666488c99717f68acfeeea4" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:10:00.658468 32416 raft_consensus.cc:2454] T 9e5188c15666488c99717f68acfeeea4 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:10:00.658530 30264 raft_consensus.cc:2454] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:10:00.658924 32489 leader_election.cc:302] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 5f8a0cfdfdca4148a75686dc88059cf3; no voters:
I20240501 14:10:00.659214 9412 raft_consensus.cc:2781] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:00.659286 9412 raft_consensus.cc:482] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:00.659327 9412 raft_consensus.cc:3037] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.660396 9620 raft_consensus.cc:482] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:00.660511 9620 raft_consensus.cc:504] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:00.661000 9620 leader_election.cc:288] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:10:00.661295 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e60603bc30424fb583de6cc7db8207" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:10:00.661420 30146 raft_consensus.cc:2454] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:10:00.661551 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e60603bc30424fb583de6cc7db8207" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:10:00.661695 30014 raft_consensus.cc:2454] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:10:00.661682 30344 leader_election.cc:302] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:10:00.661829 9620 raft_consensus.cc:2781] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:00.661876 9620 raft_consensus.cc:482] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:00.661931 9620 raft_consensus.cc:3037] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.662721 9412 raft_consensus.cc:504] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:00.663106 9412 leader_election.cc:288] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:00.663076 9620 raft_consensus.cc:504] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:00.663408 9620 leader_election.cc:288] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:10:00.663475 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5188c15666488c99717f68acfeeea4" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:10:00.663574 32416 raft_consensus.cc:3037] T 9e5188c15666488c99717f68acfeeea4 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.663655 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e60603bc30424fb583de6cc7db8207" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:10:00.663761 30146 raft_consensus.cc:3037] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.663748 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e60603bc30424fb583de6cc7db8207" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:10:00.663839 30014 raft_consensus.cc:3037] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.664789 30014 raft_consensus.cc:2454] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:10:00.664789 30146 raft_consensus.cc:2454] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:10:00.665131 30343 leader_election.cc:302] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:10:00.665288 9620 raft_consensus.cc:2781] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:00.665294 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5188c15666488c99717f68acfeeea4" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:10:00.665393 30264 raft_consensus.cc:3037] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.665434 9620 raft_consensus.cc:686] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:10:00.665571 9620 consensus_queue.cc:237] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:00.666354 30264 raft_consensus.cc:2454] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:10:00.668613 29868 catalog_manager.cc:5484] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:00.672224 32489 leader_election.cc:302] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 5f8a0cfdfdca4148a75686dc88059cf3; no voters:
I20240501 14:10:00.673044 32416 raft_consensus.cc:2454] T 9e5188c15666488c99717f68acfeeea4 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:10:00.673230 7853 raft_consensus.cc:2781] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:00.673424 7853 raft_consensus.cc:686] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:10:00.673559 7853 consensus_queue.cc:237] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:00.677939 29868 catalog_manager.cc:5484] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:10:00.686797 32533 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:00.689776 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:00.703298 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:00.703604 9729 raft_consensus.cc:482] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:00.703693 9729 raft_consensus.cc:504] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:00.704030 9729 leader_election.cc:288] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:00.704458 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f980c414f0244b19203b18a0a3f1978" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:10:00.704607 348 raft_consensus.cc:2454] T 0f980c414f0244b19203b18a0a3f1978 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:10:00.704648 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f980c414f0244b19203b18a0a3f1978" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:10:00.704793 30264 raft_consensus.cc:2454] T 0f980c414f0244b19203b18a0a3f1978 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 0.
I20240501 14:10:00.705063 32358 leader_election.cc:302] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:10:00.705317 9729 raft_consensus.cc:2781] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:00.705372 9729 raft_consensus.cc:482] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:00.705404 9729 raft_consensus.cc:3037] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.705533 30388 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:10:00.708139 29993 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:00.710949 32664 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:10:00.712199 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:00.715111 9729 raft_consensus.cc:504] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:00.715711 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f980c414f0244b19203b18a0a3f1978" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:10:00.715824 348 raft_consensus.cc:3037] T 0f980c414f0244b19203b18a0a3f1978 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.716852 348 raft_consensus.cc:2454] T 0f980c414f0244b19203b18a0a3f1978 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:10:00.717116 32358 leader_election.cc:302] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:10:00.717271 7305 raft_consensus.cc:2781] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:00.717346 7305 raft_consensus.cc:686] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:10:00.717480 7305 consensus_queue.cc:237] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:00.721055 29872 catalog_manager.cc:5484] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:10:00.731917 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f980c414f0244b19203b18a0a3f1978" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:10:00.732053 30264 raft_consensus.cc:3037] T 0f980c414f0244b19203b18a0a3f1978 P 1d0bb69611af4876a2b3b33c4766e13a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.733045 30264 raft_consensus.cc:2454] T 0f980c414f0244b19203b18a0a3f1978 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:10:00.731544 9729 leader_election.cc:288] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 1 election: Requested vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:00.744958 32401 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:00.811221 348 consensus_queue.cc:237] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:00.812525 30407 raft_consensus.cc:1261] T 5db25d87e72347f4a4410bcde440f359 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:00.812906 7382 consensus_queue.cc:1035] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:00.813380 32415 raft_consensus.cc:1261] T 5db25d87e72347f4a4410bcde440f359 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:00.813845 7382 consensus_queue.cc:1035] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:00.815732 8595 raft_consensus.cc:2932] T 5db25d87e72347f4a4410bcde440f359 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:00.817569 32416 raft_consensus.cc:2932] T 5db25d87e72347f4a4410bcde440f359 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:00.821887 29868 catalog_manager.cc:5484] T 5db25d87e72347f4a4410bcde440f359 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:00.825886 32664 tablet_service.cc:1514] Processing DeleteTablet for tablet 5db25d87e72347f4a4410bcde440f359 with delete_type TABLET_DATA_TOMBSTONED (TS db935501e9104de5b15c8780681d6c38 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35078
I20240501 14:10:00.830427 30407 raft_consensus.cc:2932] T 5db25d87e72347f4a4410bcde440f359 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:00.833576 9873 tablet_replica.cc:330] T 5db25d87e72347f4a4410bcde440f359 P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:10:00.833760 9873 raft_consensus.cc:2229] T 5db25d87e72347f4a4410bcde440f359 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:00.833909 9873 raft_consensus.cc:2258] T 5db25d87e72347f4a4410bcde440f359 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:00.834352 9873 ts_tablet_manager.cc:1875] T 5db25d87e72347f4a4410bcde440f359 P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:00.836854 9873 ts_tablet_manager.cc:1888] T 5db25d87e72347f4a4410bcde440f359 P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:10:00.836936 9873 log.cc:1196] T 5db25d87e72347f4a4410bcde440f359 P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/5db25d87e72347f4a4410bcde440f359
I20240501 14:10:00.837556 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet 5db25d87e72347f4a4410bcde440f359 (table rebalancer_extra_table_22 [id=0c280593e367426fa4c2534483cb4da4]) successfully deleted
I20240501 14:10:00.848699 29872 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:10:00.852758 30013 raft_consensus.cc:1261] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:00.853083 30146 raft_consensus.cc:1261] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:00.853202 9764 consensus_queue.cc:1035] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.853477 9764 consensus_queue.cc:1035] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.856644 29868 catalog_manager.cc:5576] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457) does not have the latest schema for tablet 36e60603bc30424fb583de6cc7db8207 (table rebalancer_dynamic_table [id=a9126bd559ea4e80ae42b4c9ad45b64d]). Expected version 1 got 0
I20240501 14:10:00.858141 9889 tablet.cc:1722] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:10:00.858285 29872 catalog_manager.cc:5576] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387) does not have the latest schema for tablet 36e60603bc30424fb583de6cc7db8207 (table rebalancer_dynamic_table [id=a9126bd559ea4e80ae42b4c9ad45b64d]). Expected version 1 got 0
I20240501 14:10:00.858491 9889 tablet.cc:1620] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7: MemRowSet was empty: no flush needed.
I20240501 14:10:00.859015 9743 mvcc.cc:205] Tried to move back new op lower bound from 7022889373088444416 to 7022889372327583744. Current Snapshot: MvccSnapshot[applied={T|T < 7022889373088444416}]
I20240501 14:10:00.859735 9890 tablet.cc:1722] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:10:00.860122 9890 tablet.cc:1620] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3: MemRowSet was empty: no flush needed.
I20240501 14:10:00.862075 29872 catalog_manager.cc:5576] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457) does not have the latest schema for tablet 36e60603bc30424fb583de6cc7db8207 (table rebalancer_dynamic_table [id=a9126bd559ea4e80ae42b4c9ad45b64d]). Expected version 1 got 0
I20240501 14:10:00.863269 9892 tablet.cc:1722] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:10:00.863535 9892 tablet.cc:1620] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e: MemRowSet was empty: no flush needed.
I20240501 14:10:00.867136 29868 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=a9126bd559ea4e80ae42b4c9ad45b64d] alter complete (version 1)
I20240501 14:10:00.867763 29872 catalog_manager.cc:5576] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387) does not have the latest schema for tablet 36e60603bc30424fb583de6cc7db8207 (table rebalancer_dynamic_table [id=a9126bd559ea4e80ae42b4c9ad45b64d]). Expected version 1 got 0
I20240501 14:10:00.907927 9244 raft_consensus.cc:482] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:00.908048 9244 raft_consensus.cc:504] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:00.908461 9244 leader_election.cc:288] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:10:00.908900 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca6bdb2e17a4e5e9dab845cad66ada8" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:10:00.909041 348 raft_consensus.cc:2454] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:10:00.909018 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca6bdb2e17a4e5e9dab845cad66ada8" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:10:00.909147 32554 raft_consensus.cc:2454] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:10:00.909335 32620 leader_election.cc:302] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:10:00.909500 9244 raft_consensus.cc:2781] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:00.909543 9244 raft_consensus.cc:482] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:00.909574 9244 raft_consensus.cc:3037] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.910552 9244 raft_consensus.cc:504] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:00.910872 9244 leader_election.cc:288] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Requested vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:10:00.911623 348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca6bdb2e17a4e5e9dab845cad66ada8" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:10:00.911731 348 raft_consensus.cc:3037] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.912475 32554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca6bdb2e17a4e5e9dab845cad66ada8" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:10:00.912591 32554 raft_consensus.cc:3037] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.912902 348 raft_consensus.cc:2454] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:10:00.913187 32620 leader_election.cc:302] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:10:00.913337 9244 raft_consensus.cc:2781] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:00.913419 9244 raft_consensus.cc:686] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Becoming Leader. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
I20240501 14:10:00.913540 9244 consensus_queue.cc:237] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:00.913659 32554 raft_consensus.cc:2454] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:10:00.919838 29864 catalog_manager.cc:5484] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db935501e9104de5b15c8780681d6c38 (127.22.252.135). New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:10:00.964776 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:00.966403 29993 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:00.969375 29864 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:10:00.970386 30388 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:10:00.973806 32664 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:00.988529 9889 tablet.cc:1722] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:10:00.988816 9889 tablet.cc:1620] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7: MemRowSet was empty: no flush needed.
I20240501 14:10:00.989540 9890 tablet.cc:1722] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:10:00.989856 9890 tablet.cc:1620] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3: MemRowSet was empty: no flush needed.
I20240501 14:10:00.990376 9892 tablet.cc:1722] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:10:00.990658 9892 tablet.cc:1620] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e: MemRowSet was empty: no flush needed.
I20240501 14:10:00.995301 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:01.007719 29864 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=a9126bd559ea4e80ae42b4c9ad45b64d] alter complete (version 2)
I20240501 14:10:01.014559 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:01.029534 32533 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:01.030931 32401 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:01.090298 348 consensus_queue.cc:237] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:01.091456 30013 raft_consensus.cc:1261] T 33de572fa35c40109cb70ee3287829f2 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:01.091537 30146 raft_consensus.cc:1261] T 33de572fa35c40109cb70ee3287829f2 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:01.091897 9871 consensus_queue.cc:1035] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:01.092056 8595 consensus_queue.cc:1035] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:01.093819 8595 raft_consensus.cc:2932] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:01.094168 30146 raft_consensus.cc:2932] T 33de572fa35c40109cb70ee3287829f2 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:01.094449 30013 raft_consensus.cc:2932] T 33de572fa35c40109cb70ee3287829f2 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:01.097534 29864 catalog_manager.cc:5484] T 33de572fa35c40109cb70ee3287829f2 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:01.101438 32401 tablet_service.cc:1514] Processing DeleteTablet for tablet 33de572fa35c40109cb70ee3287829f2 with delete_type TABLET_DATA_TOMBSTONED (TS a91029b90bf9415a89a2f70736759ad3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53090
I20240501 14:10:01.101765 9963 tablet_replica.cc:330] T 33de572fa35c40109cb70ee3287829f2 P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:10:01.101936 9963 raft_consensus.cc:2229] T 33de572fa35c40109cb70ee3287829f2 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:01.102138 9963 raft_consensus.cc:2258] T 33de572fa35c40109cb70ee3287829f2 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:01.102603 9963 ts_tablet_manager.cc:1875] T 33de572fa35c40109cb70ee3287829f2 P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:01.105049 9963 ts_tablet_manager.cc:1888] T 33de572fa35c40109cb70ee3287829f2 P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:10:01.105131 9963 log.cc:1196] T 33de572fa35c40109cb70ee3287829f2 P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/33de572fa35c40109cb70ee3287829f2
I20240501 14:10:01.105727 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 33de572fa35c40109cb70ee3287829f2 (table rebalancer_extra_table_18 [id=02ac9598abf44cafb0f816aa3165fb2c]) successfully deleted
I20240501 14:10:01.115760 29865 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:10:01.121739 9889 tablet.cc:1722] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:10:01.122828 9890 tablet.cc:1722] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:10:01.122918 9892 tablet.cc:1722] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:10:01.128134 29865 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table_new_name [id=a9126bd559ea4e80ae42b4c9ad45b64d] alter complete (version 3)
I20240501 14:10:01.169432 9838 consensus_queue.cc:1035] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:01.180713 9838 consensus_queue.cc:1035] T 9e5188c15666488c99717f68acfeeea4 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:01.199844 9459 consensus_queue.cc:1035] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:01.203109 7305 consensus_queue.cc:1035] T 0f980c414f0244b19203b18a0a3f1978 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:01.232928 29872 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:10:01.233059 29872 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:10:01.235250 29872 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: Sending DeleteTablet for 3 replicas of tablet 36e60603bc30424fb583de6cc7db8207
I20240501 14:10:01.235981 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet 36e60603bc30424fb583de6cc7db8207 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:10:01 UTC) from {username='slave'} at 127.0.0.1:46100
I20240501 14:10:01.236161 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 36e60603bc30424fb583de6cc7db8207 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:10:01 UTC) from {username='slave'} at 127.0.0.1:45302
I20240501 14:10:01.236578 30388 tablet_service.cc:1514] Processing DeleteTablet for tablet 36e60603bc30424fb583de6cc7db8207 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:10:01 UTC) from {username='slave'} at 127.0.0.1:38118
I20240501 14:10:01.237206 10012 tablet_replica.cc:330] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:10:01.237409 10012 raft_consensus.cc:2229] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:10:01.238256 10012 raft_consensus.cc:2258] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:01.238858 10012 ts_tablet_manager.cc:1875] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:10:01.240350 10011 tablet_replica.cc:330] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:10:01.240552 10011 raft_consensus.cc:2229] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:01.241457 10011 raft_consensus.cc:2258] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:01.242110 10011 ts_tablet_manager.cc:1875] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:10:01.244733 10011 ts_tablet_manager.cc:1888] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:10:01.244817 10011 log.cc:1196] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/36e60603bc30424fb583de6cc7db8207
I20240501 14:10:01.245728 10012 ts_tablet_manager.cc:1888] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:10:01.245823 10012 log.cc:1196] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/36e60603bc30424fb583de6cc7db8207
I20240501 14:10:01.246663 10012 ts_tablet_manager.cc:1909] T 36e60603bc30424fb583de6cc7db8207 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting consensus metadata
I20240501 14:10:01.247615 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 36e60603bc30424fb583de6cc7db8207 (table rebalancer_dynamic_table_new_name [id=a9126bd559ea4e80ae42b4c9ad45b64d]) successfully deleted
I20240501 14:10:01.248631 10010 tablet_replica.cc:330] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:10:01.249053 10010 raft_consensus.cc:2229] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:01.249445 10010 raft_consensus.cc:2258] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:01.249936 10011 ts_tablet_manager.cc:1909] T 36e60603bc30424fb583de6cc7db8207 P d35c16dcb506426993811bb2519d7db3: Deleting consensus metadata
I20240501 14:10:01.250200 10010 ts_tablet_manager.cc:1875] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:10:01.250828 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 36e60603bc30424fb583de6cc7db8207 (table rebalancer_dynamic_table_new_name [id=a9126bd559ea4e80ae42b4c9ad45b64d]) successfully deleted
I20240501 14:10:01.253108 10010 ts_tablet_manager.cc:1888] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:10:01.253217 10010 log.cc:1196] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/36e60603bc30424fb583de6cc7db8207
I20240501 14:10:01.253626 10010 ts_tablet_manager.cc:1909] T 36e60603bc30424fb583de6cc7db8207 P 93f2518fbadf4f9991cda93afacb363e: Deleting consensus metadata
I20240501 14:10:01.254222 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet 36e60603bc30424fb583de6cc7db8207 (table rebalancer_dynamic_table_new_name [id=a9126bd559ea4e80ae42b4c9ad45b64d]) successfully deleted
I20240501 14:10:01.273319 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:01.277761 32533 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:01.280598 32664 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:01.281190 30388 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:01.288209 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:01.292637 29993 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:01.302914 32401 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:01.329212 9925 consensus_queue.cc:1035] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:01.333890 9244 consensus_queue.cc:1035] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:01.336989 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:01.337807 29870 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:10:01.349128 29993 tablet_service.cc:1467] Processing CreateTablet for tablet cca9e338df054a4e9af4e5bfa3fef0c9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=68701843d04e4a229246524345a8aaf1]), partition=
I20240501 14:10:01.349413 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cca9e338df054a4e9af4e5bfa3fef0c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.349402 32664 tablet_service.cc:1467] Processing CreateTablet for tablet cca9e338df054a4e9af4e5bfa3fef0c9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=68701843d04e4a229246524345a8aaf1]), partition=
I20240501 14:10:01.349627 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cca9e338df054a4e9af4e5bfa3fef0c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.351979 30388 tablet_service.cc:1467] Processing CreateTablet for tablet cca9e338df054a4e9af4e5bfa3fef0c9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=68701843d04e4a229246524345a8aaf1]), partition=
I20240501 14:10:01.352102 10027 tablet_bootstrap.cc:492] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:10:01.352255 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cca9e338df054a4e9af4e5bfa3fef0c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.353143 10027 tablet_bootstrap.cc:654] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.354344 10027 tablet_bootstrap.cc:492] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:10:01.354414 10027 ts_tablet_manager.cc:1367] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:01.354643 10030 tablet_bootstrap.cc:492] T cca9e338df054a4e9af4e5bfa3fef0c9 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:10:01.355186 10027 raft_consensus.cc:348] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:01.356323 10030 tablet_bootstrap.cc:654] T cca9e338df054a4e9af4e5bfa3fef0c9 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.356472 10027 raft_consensus.cc:374] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.356582 10027 raft_consensus.cc:729] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.356773 10027 consensus_queue.cc:260] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:01.357136 10027 ts_tablet_manager.cc:1398] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:10:01.357719 10030 tablet_bootstrap.cc:492] T cca9e338df054a4e9af4e5bfa3fef0c9 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:10:01.357802 10030 ts_tablet_manager.cc:1367] T cca9e338df054a4e9af4e5bfa3fef0c9 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:10:01.358211 10030 raft_consensus.cc:348] T cca9e338df054a4e9af4e5bfa3fef0c9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:01.358372 10030 raft_consensus.cc:374] T cca9e338df054a4e9af4e5bfa3fef0c9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.358435 10030 raft_consensus.cc:729] T cca9e338df054a4e9af4e5bfa3fef0c9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.358608 10030 consensus_queue.cc:260] T cca9e338df054a4e9af4e5bfa3fef0c9 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:01.358963 10030 ts_tablet_manager.cc:1398] T cca9e338df054a4e9af4e5bfa3fef0c9 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:01.360464 10028 tablet_bootstrap.cc:492] T cca9e338df054a4e9af4e5bfa3fef0c9 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:10:01.361339 10028 tablet_bootstrap.cc:654] T cca9e338df054a4e9af4e5bfa3fef0c9 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.365352 10028 tablet_bootstrap.cc:492] T cca9e338df054a4e9af4e5bfa3fef0c9 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:10:01.365445 10028 ts_tablet_manager.cc:1367] T cca9e338df054a4e9af4e5bfa3fef0c9 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:10:01.365869 10028 raft_consensus.cc:348] T cca9e338df054a4e9af4e5bfa3fef0c9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:01.366016 10028 raft_consensus.cc:374] T cca9e338df054a4e9af4e5bfa3fef0c9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.366082 10028 raft_consensus.cc:729] T cca9e338df054a4e9af4e5bfa3fef0c9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.366236 10028 consensus_queue.cc:260] T cca9e338df054a4e9af4e5bfa3fef0c9 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:01.367318 10028 ts_tablet_manager.cc:1398] T cca9e338df054a4e9af4e5bfa3fef0c9 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:01.397265 32683 consensus_queue.cc:237] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:01.398435 30013 raft_consensus.cc:1261] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:01.398447 30406 raft_consensus.cc:1261] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:01.398813 9244 consensus_queue.cc:1035] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:01.399062 9925 consensus_queue.cc:1035] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:01.400980 9925 raft_consensus.cc:2932] T 6208e4b3d25941848b2ca533edd5a7b9 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:01.401384 30013 raft_consensus.cc:2932] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:01.404206 29872 catalog_manager.cc:5484] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:01.405385 30406 raft_consensus.cc:2932] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:01.408828 327 tablet_service.cc:1514] Processing DeleteTablet for tablet 6208e4b3d25941848b2ca533edd5a7b9 with delete_type TABLET_DATA_TOMBSTONED (TS 9ecd907cf5634ac98f3f54f3dbec6849 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45868
I20240501 14:10:01.409178 10056 tablet_replica.cc:330] T 6208e4b3d25941848b2ca533edd5a7b9 P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:10:01.409350 10056 raft_consensus.cc:2229] T 6208e4b3d25941848b2ca533edd5a7b9 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:01.409489 10056 raft_consensus.cc:2258] T 6208e4b3d25941848b2ca533edd5a7b9 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:01.410012 10056 ts_tablet_manager.cc:1875] T 6208e4b3d25941848b2ca533edd5a7b9 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:01.412408 10056 ts_tablet_manager.cc:1888] T 6208e4b3d25941848b2ca533edd5a7b9 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:10:01.412552 10056 log.cc:1196] T 6208e4b3d25941848b2ca533edd5a7b9 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/6208e4b3d25941848b2ca533edd5a7b9
I20240501 14:10:01.413110 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 6208e4b3d25941848b2ca533edd5a7b9 (table rebalancer_extra_table_48 [id=4da773d0a24f4f5d9ec6e0b50c97d669]) successfully deleted
I20240501 14:10:01.499810 29870 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47110:
name: "rebalancer_extra_table_72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:10:01.506781 32664 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2cb58cf4854a238d6f1c4bfdc64ed7 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=55a77e4e99f74942840b4808c1c6c0e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:01.507103 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2cb58cf4854a238d6f1c4bfdc64ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.507534 30388 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2cb58cf4854a238d6f1c4bfdc64ed7 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=55a77e4e99f74942840b4808c1c6c0e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:01.507768 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2cb58cf4854a238d6f1c4bfdc64ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.507714 327 tablet_service.cc:1467] Processing CreateTablet for tablet a46319e2fdff439e969092ead5d7e11e (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=55a77e4e99f74942840b4808c1c6c0e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:01.508044 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a46319e2fdff439e969092ead5d7e11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.509986 10030 tablet_bootstrap.cc:492] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:10:01.510105 30386 tablet_service.cc:1467] Processing CreateTablet for tablet 25884a00499a4861a9df4fd3a3791b2e (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=55a77e4e99f74942840b4808c1c6c0e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:01.510241 32665 tablet_service.cc:1467] Processing CreateTablet for tablet 25884a00499a4861a9df4fd3a3791b2e (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=55a77e4e99f74942840b4808c1c6c0e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:01.510334 30386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25884a00499a4861a9df4fd3a3791b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.510486 32665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25884a00499a4861a9df4fd3a3791b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.511211 10030 tablet_bootstrap.cc:654] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.511543 10028 tablet_bootstrap.cc:492] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:10:01.512565 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2cb58cf4854a238d6f1c4bfdc64ed7 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=55a77e4e99f74942840b4808c1c6c0e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:01.512660 10028 tablet_bootstrap.cc:654] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.512825 10030 tablet_bootstrap.cc:492] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:10:01.512862 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2cb58cf4854a238d6f1c4bfdc64ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.512907 10030 ts_tablet_manager.cc:1367] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:10:01.513381 10030 raft_consensus.cc:348] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:01.513495 10030 raft_consensus.cc:374] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.513531 10030 raft_consensus.cc:729] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.513634 10030 consensus_queue.cc:260] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:01.513928 10030 ts_tablet_manager.cc:1398] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:01.514089 10030 tablet_bootstrap.cc:492] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:10:01.514914 10030 tablet_bootstrap.cc:654] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.515828 30125 tablet_service.cc:1467] Processing CreateTablet for tablet a46319e2fdff439e969092ead5d7e11e (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=55a77e4e99f74942840b4808c1c6c0e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:01.516093 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a46319e2fdff439e969092ead5d7e11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.517758 10028 tablet_bootstrap.cc:492] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:10:01.517855 10028 ts_tablet_manager.cc:1367] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 14:10:01.518401 10028 raft_consensus.cc:348] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:01.518496 10028 raft_consensus.cc:374] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.518533 10028 raft_consensus.cc:729] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.518656 10028 consensus_queue.cc:260] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:01.518746 10027 tablet_bootstrap.cc:492] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:10:01.518922 10028 ts_tablet_manager.cc:1398] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:01.518862 32533 tablet_service.cc:1467] Processing CreateTablet for tablet a46319e2fdff439e969092ead5d7e11e (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=55a77e4e99f74942840b4808c1c6c0e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:01.519074 10028 tablet_bootstrap.cc:492] T 25884a00499a4861a9df4fd3a3791b2e P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:10:01.519130 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a46319e2fdff439e969092ead5d7e11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.519681 10027 tablet_bootstrap.cc:654] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P 93f2518fbadf4f9991cda93afacb363e: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.519896 10028 tablet_bootstrap.cc:654] T 25884a00499a4861a9df4fd3a3791b2e P db935501e9104de5b15c8780681d6c38: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.520597 10093 tablet_bootstrap.cc:492] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849: Bootstrap starting.
I20240501 14:10:01.521030 32534 tablet_service.cc:1467] Processing CreateTablet for tablet 25884a00499a4861a9df4fd3a3791b2e (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=55a77e4e99f74942840b4808c1c6c0e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:01.521318 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25884a00499a4861a9df4fd3a3791b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.521873 10093 tablet_bootstrap.cc:654] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.522600 10028 tablet_bootstrap.cc:492] T 25884a00499a4861a9df4fd3a3791b2e P db935501e9104de5b15c8780681d6c38: No bootstrap required, opened a new log
I20240501 14:10:01.522673 10028 ts_tablet_manager.cc:1367] T 25884a00499a4861a9df4fd3a3791b2e P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:10:01.523093 10028 raft_consensus.cc:348] T 25884a00499a4861a9df4fd3a3791b2e P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:01.523185 10028 raft_consensus.cc:374] T 25884a00499a4861a9df4fd3a3791b2e P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.523221 10028 raft_consensus.cc:729] T 25884a00499a4861a9df4fd3a3791b2e P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.523350 10028 consensus_queue.cc:260] T 25884a00499a4861a9df4fd3a3791b2e P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:01.523608 10028 ts_tablet_manager.cc:1398] T 25884a00499a4861a9df4fd3a3791b2e P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:01.529839 10101 tablet_bootstrap.cc:492] T 25884a00499a4861a9df4fd3a3791b2e P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:10:01.530681 10101 tablet_bootstrap.cc:654] T 25884a00499a4861a9df4fd3a3791b2e P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.531661 10101 tablet_bootstrap.cc:492] T 25884a00499a4861a9df4fd3a3791b2e P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:10:01.531800 10101 ts_tablet_manager.cc:1367] T 25884a00499a4861a9df4fd3a3791b2e P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:01.532330 10101 raft_consensus.cc:348] T 25884a00499a4861a9df4fd3a3791b2e P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:01.532505 10101 raft_consensus.cc:374] T 25884a00499a4861a9df4fd3a3791b2e P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.532588 10101 raft_consensus.cc:729] T 25884a00499a4861a9df4fd3a3791b2e P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.532721 10101 consensus_queue.cc:260] T 25884a00499a4861a9df4fd3a3791b2e P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:01.533021 10101 ts_tablet_manager.cc:1398] T 25884a00499a4861a9df4fd3a3791b2e P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:01.533223 10101 tablet_bootstrap.cc:492] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3: Bootstrap starting.
I20240501 14:10:01.534087 10101 tablet_bootstrap.cc:654] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.535291 10101 tablet_bootstrap.cc:492] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3: No bootstrap required, opened a new log
I20240501 14:10:01.535372 10101 ts_tablet_manager.cc:1367] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:01.535730 10101 raft_consensus.cc:348] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:10:01.535883 10101 raft_consensus.cc:374] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.535930 10101 raft_consensus.cc:729] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.536053 10101 consensus_queue.cc:260] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:10:01.536407 10101 ts_tablet_manager.cc:1398] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:01.543252 10097 tablet_bootstrap.cc:492] T a46319e2fdff439e969092ead5d7e11e P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:10:01.544718 10030 tablet_bootstrap.cc:492] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7: No bootstrap required, opened a new log
I20240501 14:10:01.544826 10030 ts_tablet_manager.cc:1367] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.001s
I20240501 14:10:01.545248 10030 raft_consensus.cc:348] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:01.545352 10030 raft_consensus.cc:374] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.545384 10030 raft_consensus.cc:729] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.545486 10030 consensus_queue.cc:260] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:01.545742 10030 ts_tablet_manager.cc:1398] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:01.546222 10027 tablet_bootstrap.cc:492] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P 93f2518fbadf4f9991cda93afacb363e: No bootstrap required, opened a new log
I20240501 14:10:01.546319 10027 ts_tablet_manager.cc:1367] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20240501 14:10:01.546718 10027 raft_consensus.cc:348] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:01.546830 10027 raft_consensus.cc:374] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.546860 10027 raft_consensus.cc:729] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.546989 10027 consensus_queue.cc:260] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:01.547278 10027 ts_tablet_manager.cc:1398] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:01.551158 10097 tablet_bootstrap.cc:654] T a46319e2fdff439e969092ead5d7e11e P d35c16dcb506426993811bb2519d7db3: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.553051 10093 tablet_bootstrap.cc:492] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849: No bootstrap required, opened a new log
I20240501 14:10:01.553202 10093 ts_tablet_manager.cc:1367] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20240501 14:10:01.553813 10097 tablet_bootstrap.cc:492] T a46319e2fdff439e969092ead5d7e11e P d35c16dcb506426993811bb2519d7db3: No bootstrap required, opened a new log
I20240501 14:10:01.553905 10097 ts_tablet_manager.cc:1367] T a46319e2fdff439e969092ead5d7e11e P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20240501 14:10:01.553864 10093 raft_consensus.cc:348] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:10:01.553959 10093 raft_consensus.cc:374] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.554005 10093 raft_consensus.cc:729] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.554265 10093 consensus_queue.cc:260] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:10:01.554626 10093 ts_tablet_manager.cc:1398] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:01.554643 10097 raft_consensus.cc:348] T a46319e2fdff439e969092ead5d7e11e P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:10:01.554754 10097 raft_consensus.cc:374] T a46319e2fdff439e969092ead5d7e11e P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.554788 10097 raft_consensus.cc:729] T a46319e2fdff439e969092ead5d7e11e P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.554900 10097 consensus_queue.cc:260] T a46319e2fdff439e969092ead5d7e11e P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:10:01.555174 10097 ts_tablet_manager.cc:1398] T a46319e2fdff439e969092ead5d7e11e P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:01.590250 9925 raft_consensus.cc:482] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:01.590339 9925 raft_consensus.cc:504] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:01.590663 9925 leader_election.cc:288] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:10:01.590970 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2cb58cf4854a238d6f1c4bfdc64ed7" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:10:01.591077 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2cb58cf4854a238d6f1c4bfdc64ed7" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:10:01.591131 30406 raft_consensus.cc:2454] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:10:01.591194 30014 raft_consensus.cc:2454] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 0.
I20240501 14:10:01.591423 32618 leader_election.cc:302] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:10:01.591573 9925 raft_consensus.cc:2781] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:01.591629 9925 raft_consensus.cc:482] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:01.591658 9925 raft_consensus.cc:3037] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.592123 9412 raft_consensus.cc:482] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:01.592198 9412 raft_consensus.cc:504] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:10:01.592567 9412 leader_election.cc:288] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:10:01.592720 9925 raft_consensus.cc:504] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:01.593091 9925 leader_election.cc:288] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Requested vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:10:01.593430 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2cb58cf4854a238d6f1c4bfdc64ed7" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:10:01.593530 30406 raft_consensus.cc:3037] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.593948 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46319e2fdff439e969092ead5d7e11e" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:10:01.594081 349 raft_consensus.cc:2454] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:10:01.594506 30406 raft_consensus.cc:2454] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:10:01.594853 32618 leader_election.cc:302] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:10:01.594949 32489 leader_election.cc:302] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:10:01.595145 9412 raft_consensus.cc:2781] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:01.595191 9412 raft_consensus.cc:482] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:01.595222 9412 raft_consensus.cc:3037] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.595799 9925 raft_consensus.cc:2781] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:01.596294 9925 raft_consensus.cc:686] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Becoming Leader. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
I20240501 14:10:01.596428 9925 consensus_queue.cc:237] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:01.596676 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2cb58cf4854a238d6f1c4bfdc64ed7" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:10:01.596796 30014 raft_consensus.cc:3037] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.597837 30014 raft_consensus.cc:2454] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:10:01.599175 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46319e2fdff439e969092ead5d7e11e" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:10:01.599313 30146 raft_consensus.cc:2454] T a46319e2fdff439e969092ead5d7e11e P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 0.
I20240501 14:10:01.600502 29866 catalog_manager.cc:5484] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db935501e9104de5b15c8780681d6c38 (127.22.252.135). New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:01.601696 9412 raft_consensus.cc:504] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:10:01.602102 9412 leader_election.cc:288] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:10:01.602773 30146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46319e2fdff439e969092ead5d7e11e" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:10:01.602895 30146 raft_consensus.cc:3037] T a46319e2fdff439e969092ead5d7e11e P d35c16dcb506426993811bb2519d7db3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.603845 30146 raft_consensus.cc:2454] T a46319e2fdff439e969092ead5d7e11e P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:10:01.604139 32490 leader_election.cc:302] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:10:01.604302 9412 raft_consensus.cc:2781] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:01.604408 9412 raft_consensus.cc:686] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8a0cfdfdca4148a75686dc88059cf3, State: Running, Role: LEADER
I20240501 14:10:01.604697 9412 consensus_queue.cc:237] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:10:01.605614 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46319e2fdff439e969092ead5d7e11e" candidate_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:10:01.605729 349 raft_consensus.cc:3037] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.606730 349 raft_consensus.cc:2454] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8a0cfdfdca4148a75686dc88059cf3 in term 1.
I20240501 14:10:01.607447 29869 catalog_manager.cc:5484] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134). New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:10:01.755954 9764 raft_consensus.cc:482] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:01.756059 9764 raft_consensus.cc:504] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:01.756434 9764 leader_election.cc:288] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:10:01.756688 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25884a00499a4861a9df4fd3a3791b2e" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:10:01.756739 9765 raft_consensus.cc:482] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:01.756736 32552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25884a00499a4861a9df4fd3a3791b2e" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" is_pre_election: true
I20240501 14:10:01.756829 32683 raft_consensus.cc:2454] T 25884a00499a4861a9df4fd3a3791b2e P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:10:01.756832 9765 raft_consensus.cc:504] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:01.756876 32552 raft_consensus.cc:2454] T 25884a00499a4861a9df4fd3a3791b2e P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 0.
I20240501 14:10:01.757125 30344 leader_election.cc:302] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:10:01.757208 9765 leader_election.cc:288] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:10:01.757470 9764 raft_consensus.cc:2781] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:01.757530 9764 raft_consensus.cc:482] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:01.757539 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca9e338df054a4e9af4e5bfa3fef0c9" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:10:01.757577 9764 raft_consensus.cc:3037] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.757637 32683 raft_consensus.cc:2454] T cca9e338df054a4e9af4e5bfa3fef0c9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:10:01.757833 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca9e338df054a4e9af4e5bfa3fef0c9" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:10:01.757858 29948 leader_election.cc:302] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:10:01.757920 30406 raft_consensus.cc:2454] T cca9e338df054a4e9af4e5bfa3fef0c9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 0.
I20240501 14:10:01.758025 9765 raft_consensus.cc:2781] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:01.758076 9765 raft_consensus.cc:482] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:01.758111 9765 raft_consensus.cc:3037] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.758682 9764 raft_consensus.cc:504] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:01.758914 9765 raft_consensus.cc:504] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:01.759255 9765 leader_election.cc:288] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:10:01.758935 9764 leader_election.cc:288] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:10:01.759229 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25884a00499a4861a9df4fd3a3791b2e" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:10:01.759296 32552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25884a00499a4861a9df4fd3a3791b2e" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:10:01.759338 32683 raft_consensus.cc:3037] T 25884a00499a4861a9df4fd3a3791b2e P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.759435 32552 raft_consensus.cc:3037] T 25884a00499a4861a9df4fd3a3791b2e P 5f8a0cfdfdca4148a75686dc88059cf3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.759639 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca9e338df054a4e9af4e5bfa3fef0c9" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:10:01.759721 32684 raft_consensus.cc:3037] T cca9e338df054a4e9af4e5bfa3fef0c9 P db935501e9104de5b15c8780681d6c38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.759688 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca9e338df054a4e9af4e5bfa3fef0c9" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:10:01.759810 30406 raft_consensus.cc:3037] T cca9e338df054a4e9af4e5bfa3fef0c9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.760457 32552 raft_consensus.cc:2454] T 25884a00499a4861a9df4fd3a3791b2e P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:10:01.760643 32684 raft_consensus.cc:2454] T cca9e338df054a4e9af4e5bfa3fef0c9 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:10:01.760704 30406 raft_consensus.cc:2454] T cca9e338df054a4e9af4e5bfa3fef0c9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:10:01.760758 30344 leader_election.cc:302] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8a0cfdfdca4148a75686dc88059cf3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:10:01.761010 9764 raft_consensus.cc:2781] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:01.761096 9764 raft_consensus.cc:686] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:10:01.761139 29948 leader_election.cc:302] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:10:01.761245 9764 consensus_queue.cc:237] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:01.761443 9765 raft_consensus.cc:2781] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:01.761509 9765 raft_consensus.cc:686] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:10:01.761610 9765 consensus_queue.cc:237] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:01.762770 32683 raft_consensus.cc:2454] T 25884a00499a4861a9df4fd3a3791b2e P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:10:01.764117 29866 catalog_manager.cc:5484] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:01.765339 29869 catalog_manager.cc:5484] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:01.794055 32533 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:01.794714 32401 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:01.806244 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:01.809487 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:01.815275 30386 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:10:01.834744 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:01.837932 29993 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:01.856873 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:01.913820 29869 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:10:01.915558 30406 consensus_queue.cc:237] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:10:01.916759 32552 raft_consensus.cc:1261] T cfc701c19cd74d0db2f90ce580230177 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:01.916838 349 raft_consensus.cc:1261] T cfc701c19cd74d0db2f90ce580230177 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:01.917137 9937 consensus_queue.cc:1035] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:01.917361 9764 consensus_queue.cc:1035] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:01.919832 9937 raft_consensus.cc:2932] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:10:01.920318 32552 raft_consensus.cc:2932] T cfc701c19cd74d0db2f90ce580230177 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:10:01.921752 349 raft_consensus.cc:2932] T cfc701c19cd74d0db2f90ce580230177 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:10:01.921926 32683 raft_consensus.cc:1261] T cca9e338df054a4e9af4e5bfa3fef0c9 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:01.922343 7378 consensus_queue.cc:1035] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:01.923386 29869 catalog_manager.cc:5484] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index 6 to 7, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:10:01.923789 30406 raft_consensus.cc:1261] T cca9e338df054a4e9af4e5bfa3fef0c9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:01.924206 7378 consensus_queue.cc:1035] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:01.927604 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet cfc701c19cd74d0db2f90ce580230177 with delete_type TABLET_DATA_TOMBSTONED (TS d35c16dcb506426993811bb2519d7db3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45302
I20240501 14:10:01.927964 10145 tablet_replica.cc:330] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:10:01.928273 10145 raft_consensus.cc:2229] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:01.928582 10145 raft_consensus.cc:2258] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:01.929273 10145 ts_tablet_manager.cc:1875] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:01.929921 29867 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet cca9e338df054a4e9af4e5bfa3fef0c9 (table rebalancer_dynamic_table [id=68701843d04e4a229246524345a8aaf1]). Expected version 1 got 0
I20240501 14:10:01.932132 10145 ts_tablet_manager.cc:1888] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:10:01.932269 10145 log.cc:1196] T cfc701c19cd74d0db2f90ce580230177 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/cfc701c19cd74d0db2f90ce580230177
I20240501 14:10:01.932775 10154 tablet.cc:1722] T cca9e338df054a4e9af4e5bfa3fef0c9 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:10:01.933108 10154 tablet.cc:1620] T cca9e338df054a4e9af4e5bfa3fef0c9 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:10:01.933113 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet cfc701c19cd74d0db2f90ce580230177 (table rebalance_test_table_1 [id=df055757a36d4b9f89886608b024f0b8]) successfully deleted
I20240501 14:10:01.934906 29869 catalog_manager.cc:5576] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915) does not have the latest schema for tablet cca9e338df054a4e9af4e5bfa3fef0c9 (table rebalancer_dynamic_table [id=68701843d04e4a229246524345a8aaf1]). Expected version 1 got 0
I20240501 14:10:01.936801 29867 catalog_manager.cc:5576] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357) does not have the latest schema for tablet cca9e338df054a4e9af4e5bfa3fef0c9 (table rebalancer_dynamic_table [id=68701843d04e4a229246524345a8aaf1]). Expected version 1 got 0
I20240501 14:10:01.938488 10151 tablet.cc:1722] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:10:01.938808 10151 tablet.cc:1620] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e: MemRowSet was empty: no flush needed.
I20240501 14:10:01.939530 29869 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=68701843d04e4a229246524345a8aaf1] alter complete (version 1)
I20240501 14:10:01.940351 10155 tablet.cc:1722] T cca9e338df054a4e9af4e5bfa3fef0c9 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:10:01.940804 10155 tablet.cc:1620] T cca9e338df054a4e9af4e5bfa3fef0c9 P d1c217469ae4478ebd49e4ed9f7352d7: MemRowSet was empty: no flush needed.
I20240501 14:10:01.942377 29869 catalog_manager.cc:5576] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357) does not have the latest schema for tablet cca9e338df054a4e9af4e5bfa3fef0c9 (table rebalancer_dynamic_table [id=68701843d04e4a229246524345a8aaf1]). Expected version 1 got 0
I20240501 14:10:02.046226 29866 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47110:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:10:02.053232 10151 tablet.cc:1722] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:10:02.053560 10151 tablet.cc:1620] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e: MemRowSet was empty: no flush needed.
I20240501 14:10:02.053577 10154 tablet.cc:1722] T cca9e338df054a4e9af4e5bfa3fef0c9 P db935501e9104de5b15c8780681d6c38: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:10:02.053848 10154 tablet.cc:1620] T cca9e338df054a4e9af4e5bfa3fef0c9 P db935501e9104de5b15c8780681d6c38: MemRowSet was empty: no flush needed.
I20240501 14:10:02.054329 10155 tablet.cc:1722] T cca9e338df054a4e9af4e5bfa3fef0c9 P d1c217469ae4478ebd49e4ed9f7352d7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:10:02.054596 10155 tablet.cc:1620] T cca9e338df054a4e9af4e5bfa3fef0c9 P d1c217469ae4478ebd49e4ed9f7352d7: MemRowSet was empty: no flush needed.
I20240501 14:10:02.064823 29866 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f2cd510126964a349f8d7712a2b33cf5: rebalancer_dynamic_table [id=68701843d04e4a229246524345a8aaf1] alter complete (version 2)
I20240501 14:10:02.096787 29993 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:02.096751 9925 consensus_queue.cc:1035] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:02.100554 32533 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:02.102986 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:02.105247 32401 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:02.118427 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:02.123551 10005 consensus_queue.cc:1035] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:02.129717 30386 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:10:02.138963 9412 consensus_queue.cc:1035] T a46319e2fdff439e969092ead5d7e11e P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:02.143981 9244 consensus_queue.cc:1035] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:02.160193 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:02.160869 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:02.195452 9764 consensus_queue.cc:1035] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:02.211148 9764 consensus_queue.cc:1035] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:02.225922 30407 consensus_queue.cc:237] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:10:02.226982 32683 raft_consensus.cc:1261] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:10:02.226938 32552 raft_consensus.cc:1261] T c362bd01a59b4bcf85eeb59713e58b64 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 2 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:10:02.227357 10142 consensus_queue.cc:1035] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:02.227509 9764 consensus_queue.cc:1035] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:02.229223 9764 raft_consensus.cc:2932] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:10:02.229805 32683 raft_consensus.cc:2932] T c362bd01a59b4bcf85eeb59713e58b64 P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:10:02.229954 32552 raft_consensus.cc:2932] T c362bd01a59b4bcf85eeb59713e58b64 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:10:02.233367 29865 catalog_manager.cc:5484] T c362bd01a59b4bcf85eeb59713e58b64 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: config changed from index 7 to 8, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New cstate: current_term: 2 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:10:02.238099 30255 tablet_service.cc:1514] Processing DeleteTablet for tablet c362bd01a59b4bcf85eeb59713e58b64 with delete_type TABLET_DATA_TOMBSTONED (TS 1d0bb69611af4876a2b3b33c4766e13a not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:44464
I20240501 14:10:02.238448 10237 tablet_replica.cc:330] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:10:02.238622 10237 raft_consensus.cc:2229] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:02.239487 10237 raft_consensus.cc:2258] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:02.240044 10237 ts_tablet_manager.cc:1875] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:02.242694 10237 ts_tablet_manager.cc:1888] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240501 14:10:02.242761 10237 log.cc:1196] T c362bd01a59b4bcf85eeb59713e58b64 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/c362bd01a59b4bcf85eeb59713e58b64
I20240501 14:10:02.243322 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet c362bd01a59b4bcf85eeb59713e58b64 (table rebalance_test_table_13 [id=422856ed458a4c52950e8e3e49019a55]) successfully deleted
I20240501 14:10:02.395442 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:02.396364 32533 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:02.419864 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:02.420647 30386 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:10:02.427810 29993 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:02.428392 32401 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:02.437640 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:02.448419 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:02.508826 32684 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8bb9687045d44efa888a69c7acde1105"
dest_uuid: "db935501e9104de5b15c8780681d6c38"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42724
I20240501 14:10:02.509002 32684 raft_consensus.cc:595] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:10:02.522173 9244 raft_consensus.cc:979] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38: : Instructing follower d1c217469ae4478ebd49e4ed9f7352d7 to start an election
I20240501 14:10:02.522284 9244 raft_consensus.cc:1067] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Signalling peer d1c217469ae4478ebd49e4ed9f7352d7 to start an election
I20240501 14:10:02.523308 30407 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8bb9687045d44efa888a69c7acde1105"
dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
from {username='slave'} at 127.22.252.135:43131
I20240501 14:10:02.523434 30407 raft_consensus.cc:482] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:10:02.523487 30407 raft_consensus.cc:3037] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:02.524672 30407 raft_consensus.cc:504] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:02.525117 30407 leader_election.cc:288] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:10:02.525405 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb9687045d44efa888a69c7acde1105" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:10:02.525540 349 raft_consensus.cc:3037] T 8bb9687045d44efa888a69c7acde1105 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:02.526628 349 raft_consensus.cc:2454] T 8bb9687045d44efa888a69c7acde1105 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 2.
I20240501 14:10:02.526612 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb9687045d44efa888a69c7acde1105" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:10:02.526717 32684 raft_consensus.cc:3032] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:10:02.526755 32684 raft_consensus.cc:729] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
I20240501 14:10:02.526895 32684 consensus_queue.cc:260] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:02.527104 32684 raft_consensus.cc:3037] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:02.528093 32684 raft_consensus.cc:2454] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 2.
I20240501 14:10:02.528508 30341 leader_election.cc:302] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, d1c217469ae4478ebd49e4ed9f7352d7, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:10:02.528656 9937 raft_consensus.cc:2781] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:02.528740 9937 raft_consensus.cc:686] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:10:02.528869 9937 consensus_queue.cc:237] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:02.529906 32552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb9687045d44efa888a69c7acde1105" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "5f8a0cfdfdca4148a75686dc88059cf3"
I20240501 14:10:02.530037 32552 raft_consensus.cc:3037] T 8bb9687045d44efa888a69c7acde1105 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:02.531127 32552 raft_consensus.cc:2454] T 8bb9687045d44efa888a69c7acde1105 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 2.
I20240501 14:10:02.533651 29870 catalog_manager.cc:5484] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 1 to 2, leader changed from db935501e9104de5b15c8780681d6c38 (127.22.252.135) to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 2 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:02.689640 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:02.692157 32401 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:02.692962 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:02.693840 29993 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:02.696715 32533 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:02.710196 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:02.717689 32665 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:02.718698 30386 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:02.906523 32533 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:02.910585 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:02.911077 29993 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:02.912086 30386 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:02.920331 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:02.931496 32554 raft_consensus.cc:1261] T 8bb9687045d44efa888a69c7acde1105 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 2 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:10:02.932053 9937 consensus_queue.cc:1035] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:02.932746 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:02.938766 32665 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:02.958542 348 raft_consensus.cc:1261] T 8bb9687045d44efa888a69c7acde1105 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:10:02.958916 9937 consensus_queue.cc:1035] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:02.969288 32683 raft_consensus.cc:1261] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:10:02.969319 32401 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:02.969823 10394 consensus_queue.cc:1035] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 14:10:03.033149 30145 consensus_queue.cc:237] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:03.034449 30013 raft_consensus.cc:1261] T c7df6b8d76714eebbeacc85f403d3405 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:03.034741 32554 raft_consensus.cc:1261] T c7df6b8d76714eebbeacc85f403d3405 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:10:03.034802 7205 consensus_queue.cc:1035] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:03.035279 7205 consensus_queue.cc:1035] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:10:03.037298 7205 raft_consensus.cc:2932] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:03.037575 32554 raft_consensus.cc:2932] T c7df6b8d76714eebbeacc85f403d3405 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:03.037892 30013 raft_consensus.cc:2932] T c7df6b8d76714eebbeacc85f403d3405 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:03.041034 29864 catalog_manager.cc:5484] T c7df6b8d76714eebbeacc85f403d3405 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: config changed from index 4 to 5, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New cstate: current_term: 1 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:03.046092 32401 tablet_service.cc:1514] Processing DeleteTablet for tablet c7df6b8d76714eebbeacc85f403d3405 with delete_type TABLET_DATA_TOMBSTONED (TS a91029b90bf9415a89a2f70736759ad3 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:53090
I20240501 14:10:03.046458 10426 tablet_replica.cc:330] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:10:03.047748 10426 raft_consensus.cc:2229] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:03.047966 10426 raft_consensus.cc:2258] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:03.051589 10426 ts_tablet_manager.cc:1875] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:03.053953 10426 ts_tablet_manager.cc:1888] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:10:03.054028 10426 log.cc:1196] T c7df6b8d76714eebbeacc85f403d3405 P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/c7df6b8d76714eebbeacc85f403d3405
I20240501 14:10:03.054512 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet c7df6b8d76714eebbeacc85f403d3405 (table rebalancer_extra_table_8 [id=8ff8f244c547410f853c771c11ecdd1e]) successfully deleted
I20240501 14:10:03.201298 29993 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:03.206957 32533 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:03.223407 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:03.224668 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:03.227597 32665 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:03.231040 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:03.231201 32401 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:03.251142 30386 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:03.313061 32684 consensus_queue.cc:237] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:03.314296 32552 raft_consensus.cc:1261] T 0cccc8eec30141c1946b57d3be27000f P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:03.314397 30014 raft_consensus.cc:1261] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:03.314606 9925 consensus_queue.cc:1035] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:03.314810 10474 consensus_queue.cc:1035] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:03.316860 9925 raft_consensus.cc:2932] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:03.317588 30014 raft_consensus.cc:2932] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:03.318646 32552 raft_consensus.cc:2932] T 0cccc8eec30141c1946b57d3be27000f P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:03.321180 29867 catalog_manager.cc:5484] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e reported cstate change: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:03.324527 327 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cccc8eec30141c1946b57d3be27000f with delete_type TABLET_DATA_TOMBSTONED (TS 9ecd907cf5634ac98f3f54f3dbec6849 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45868
I20240501 14:10:03.324878 10493 tablet_replica.cc:330] T 0cccc8eec30141c1946b57d3be27000f P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:10:03.326632 10493 raft_consensus.cc:2229] T 0cccc8eec30141c1946b57d3be27000f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:03.326793 10493 raft_consensus.cc:2258] T 0cccc8eec30141c1946b57d3be27000f P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:03.327275 10493 ts_tablet_manager.cc:1875] T 0cccc8eec30141c1946b57d3be27000f P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:03.329803 10493 ts_tablet_manager.cc:1888] T 0cccc8eec30141c1946b57d3be27000f P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:10:03.329864 10493 log.cc:1196] T 0cccc8eec30141c1946b57d3be27000f P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/0cccc8eec30141c1946b57d3be27000f
I20240501 14:10:03.330353 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 0cccc8eec30141c1946b57d3be27000f (table rebalancer_extra_table_14 [id=c6639c36fcc54bb09704a0d7c258496b]) successfully deleted
I20240501 14:10:03.451881 32533 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:03.455931 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:03.463486 29993 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:03.476068 30386 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:03.479288 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:03.487262 32665 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:03.507827 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:03.508663 32401 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:03.562515 30264 consensus_queue.cc:237] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:03.563642 30145 raft_consensus.cc:1261] T a4462d7eaaa54223998369121761acd6 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:03.564038 349 raft_consensus.cc:1261] T a4462d7eaaa54223998369121761acd6 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:03.564128 8437 consensus_queue.cc:1035] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:03.564453 9823 consensus_queue.cc:1035] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:03.566140 9823 raft_consensus.cc:2932] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:03.566439 30146 raft_consensus.cc:2932] T a4462d7eaaa54223998369121761acd6 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:03.566663 349 raft_consensus.cc:2932] T a4462d7eaaa54223998369121761acd6 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:03.569599 29869 catalog_manager.cc:5484] T a4462d7eaaa54223998369121761acd6 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: config changed from index 2 to 3, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) evicted. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:03.572827 32533 tablet_service.cc:1514] Processing DeleteTablet for tablet a4462d7eaaa54223998369121761acd6 with delete_type TABLET_DATA_TOMBSTONED (TS 5f8a0cfdfdca4148a75686dc88059cf3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47638
I20240501 14:10:03.573237 10565 tablet_replica.cc:330] T a4462d7eaaa54223998369121761acd6 P 5f8a0cfdfdca4148a75686dc88059cf3: stopping tablet replica
I20240501 14:10:03.573408 10565 raft_consensus.cc:2229] T a4462d7eaaa54223998369121761acd6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:03.573596 10565 raft_consensus.cc:2258] T a4462d7eaaa54223998369121761acd6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:03.574095 10565 ts_tablet_manager.cc:1875] T a4462d7eaaa54223998369121761acd6 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:03.576490 10565 ts_tablet_manager.cc:1888] T a4462d7eaaa54223998369121761acd6 P 5f8a0cfdfdca4148a75686dc88059cf3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:10:03.576565 10565 log.cc:1196] T a4462d7eaaa54223998369121761acd6 P 5f8a0cfdfdca4148a75686dc88059cf3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/wals/a4462d7eaaa54223998369121761acd6
I20240501 14:10:03.577132 29860 catalog_manager.cc:4830] TS 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): tablet a4462d7eaaa54223998369121761acd6 (table rebalancer_extra_table_56 [id=8f130519999a4055a609c067e3e6579f]) successfully deleted
I20240501 14:10:03.716770 29993 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:03.717947 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:03.721395 32533 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:03.723310 30386 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:03.743146 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:03.744263 32401 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:03.745292 32665 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:03.758061 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:03.848528 30407 consensus_queue.cc:237] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:10:03.849951 349 raft_consensus.cc:1261] T dd2fd47ec6684cb28d2f4678d2e90e75 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:03.850006 32684 raft_consensus.cc:1261] T dd2fd47ec6684cb28d2f4678d2e90e75 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:03.850463 10394 consensus_queue.cc:1035] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:03.850679 9764 consensus_queue.cc:1035] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:03.852794 10394 raft_consensus.cc:2932] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:10:03.856911 349 raft_consensus.cc:2932] T dd2fd47ec6684cb28d2f4678d2e90e75 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:10:03.862560 29873 catalog_manager.cc:5484] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index 6 to 7, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:10:03.865335 30255 tablet_service.cc:1514] Processing DeleteTablet for tablet dd2fd47ec6684cb28d2f4678d2e90e75 with delete_type TABLET_DATA_TOMBSTONED (TS 1d0bb69611af4876a2b3b33c4766e13a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44464
I20240501 14:10:03.865757 10641 tablet_replica.cc:330] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:10:03.865878 10641 raft_consensus.cc:2229] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:03.866015 10641 raft_consensus.cc:2258] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:03.866470 10641 ts_tablet_manager.cc:1875] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:03.867429 32684 raft_consensus.cc:2932] T dd2fd47ec6684cb28d2f4678d2e90e75 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:10:03.870380 10641 ts_tablet_manager.cc:1888] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:10:03.870597 10641 log.cc:1196] T dd2fd47ec6684cb28d2f4678d2e90e75 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/dd2fd47ec6684cb28d2f4678d2e90e75
I20240501 14:10:03.871773 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet dd2fd47ec6684cb28d2f4678d2e90e75 (table rebalance_test_table_17 [id=9a4b9a0a2fe64ab2a64d6679185c2a33]) successfully deleted
I20240501 14:10:04.176694 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:04.178671 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:04.182538 29993 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:04.187376 30386 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:04.196702 32533 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:04.209390 32401 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:04.235246 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:04.245160 32665 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:04.293022 30014 consensus_queue.cc:237] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:04.294366 32415 raft_consensus.cc:1261] T 14267ff763f5409998f457e14e407ed0 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:04.294446 30407 raft_consensus.cc:1261] T 14267ff763f5409998f457e14e407ed0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:04.294741 10192 consensus_queue.cc:1035] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:04.294930 10694 consensus_queue.cc:1035] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:04.296646 10724 raft_consensus.cc:2932] T 14267ff763f5409998f457e14e407ed0 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:04.297083 30407 raft_consensus.cc:2932] T 14267ff763f5409998f457e14e407ed0 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:04.297080 32415 raft_consensus.cc:2932] T 14267ff763f5409998f457e14e407ed0 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:04.300390 29873 catalog_manager.cc:5484] T 14267ff763f5409998f457e14e407ed0 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:04.304009 32665 tablet_service.cc:1514] Processing DeleteTablet for tablet 14267ff763f5409998f457e14e407ed0 with delete_type TABLET_DATA_TOMBSTONED (TS db935501e9104de5b15c8780681d6c38 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35078
I20240501 14:10:04.304404 10725 tablet_replica.cc:330] T 14267ff763f5409998f457e14e407ed0 P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:10:04.304576 10725 raft_consensus.cc:2229] T 14267ff763f5409998f457e14e407ed0 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:04.304862 10725 raft_consensus.cc:2258] T 14267ff763f5409998f457e14e407ed0 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:04.305341 10725 ts_tablet_manager.cc:1875] T 14267ff763f5409998f457e14e407ed0 P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:04.307302 10725 ts_tablet_manager.cc:1888] T 14267ff763f5409998f457e14e407ed0 P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:10:04.307381 10725 log.cc:1196] T 14267ff763f5409998f457e14e407ed0 P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/14267ff763f5409998f457e14e407ed0
I20240501 14:10:04.307868 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet 14267ff763f5409998f457e14e407ed0 (table rebalancer_extra_table_38 [id=b8abb09b9b7a47a3a002b43c042f0525]) successfully deleted
I20240501 14:10:04.660063 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:04.660954 29993 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:04.661917 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:04.675645 32533 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:04.689630 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:04.689657 32401 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:04.690965 30386 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:04.693051 32665 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:04.757535 32684 consensus_queue.cc:237] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:04.758639 32554 raft_consensus.cc:1261] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:04.758821 30013 raft_consensus.cc:1261] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:04.759002 9244 consensus_queue.cc:1035] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:04.759181 10647 consensus_queue.cc:1035] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:04.760803 10798 raft_consensus.cc:2932] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:04.761140 32554 raft_consensus.cc:2932] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:04.761140 30013 raft_consensus.cc:2932] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:04.764545 29870 catalog_manager.cc:5484] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 reported cstate change: config changed from index 2 to 3, VOTER 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136) evicted. New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:04.767844 327 tablet_service.cc:1514] Processing DeleteTablet for tablet 65bc0e6ad1fa491e8e5ae225f86561f9 with delete_type TABLET_DATA_TOMBSTONED (TS 9ecd907cf5634ac98f3f54f3dbec6849 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45868
I20240501 14:10:04.768294 10799 tablet_replica.cc:330] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 9ecd907cf5634ac98f3f54f3dbec6849: stopping tablet replica
I20240501 14:10:04.768478 10799 raft_consensus.cc:2229] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:04.768839 10799 raft_consensus.cc:2258] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:04.769460 10799 ts_tablet_manager.cc:1875] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:04.771536 10799 ts_tablet_manager.cc:1888] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 9ecd907cf5634ac98f3f54f3dbec6849: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:10:04.771601 10799 log.cc:1196] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 9ecd907cf5634ac98f3f54f3dbec6849: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/wals/65bc0e6ad1fa491e8e5ae225f86561f9
I20240501 14:10:04.772068 29859 catalog_manager.cc:4830] TS 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937): tablet 65bc0e6ad1fa491e8e5ae225f86561f9 (table rebalancer_extra_table_16 [id=6139b5f2fcc94630ac7d807b61ccd1ff]) successfully deleted
I20240501 14:10:04.910573 32533 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:04.911901 29993 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:04.916661 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:04.920185 30386 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:04.922487 32401 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:04.922734 32665 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:04.923135 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:04.940681 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:05.010984 30407 consensus_queue.cc:237] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:05.012173 32554 raft_consensus.cc:1261] T 28500f605e0644b69207077f57023ca6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:05.012178 30146 raft_consensus.cc:1261] T 28500f605e0644b69207077f57023ca6 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:05.012563 9764 consensus_queue.cc:1035] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:05.012771 10640 consensus_queue.cc:1035] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:05.014364 10880 raft_consensus.cc:2932] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:05.014828 30146 raft_consensus.cc:2932] T 28500f605e0644b69207077f57023ca6 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:05.014910 32554 raft_consensus.cc:2932] T 28500f605e0644b69207077f57023ca6 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:05.018190 29865 catalog_manager.cc:5484] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:05.022297 32401 tablet_service.cc:1514] Processing DeleteTablet for tablet 28500f605e0644b69207077f57023ca6 with delete_type TABLET_DATA_TOMBSTONED (TS a91029b90bf9415a89a2f70736759ad3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53090
I20240501 14:10:05.022635 10886 tablet_replica.cc:330] T 28500f605e0644b69207077f57023ca6 P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:10:05.022817 10886 raft_consensus.cc:2229] T 28500f605e0644b69207077f57023ca6 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:05.022949 10886 raft_consensus.cc:2258] T 28500f605e0644b69207077f57023ca6 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:05.023408 10886 ts_tablet_manager.cc:1875] T 28500f605e0644b69207077f57023ca6 P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:05.025817 10886 ts_tablet_manager.cc:1888] T 28500f605e0644b69207077f57023ca6 P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:10:05.025902 10886 log.cc:1196] T 28500f605e0644b69207077f57023ca6 P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/28500f605e0644b69207077f57023ca6
I20240501 14:10:05.026525 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 28500f605e0644b69207077f57023ca6 (table rebalancer_extra_table_36 [id=f4bc8404fe4a4d7a89d0369621ce5f0e]) successfully deleted
I20240501 14:10:05.199949 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:05.207610 30386 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:05.225719 32665 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:05.229487 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:05.229533 29993 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:05.230443 32401 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:05.244629 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:05.255179 32533 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:05.308424 32416 consensus_queue.cc:237] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:05.309795 32554 raft_consensus.cc:1261] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 2 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:10:05.309830 30277 raft_consensus.cc:1261] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:10:05.310276 9729 consensus_queue.cc:1035] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:05.310489 10391 consensus_queue.cc:1035] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:05.312415 10391 raft_consensus.cc:2932] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:05.312772 30277 raft_consensus.cc:2932] T 5518112449d9448e98f5c779f1fa3248 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:05.314393 32554 raft_consensus.cc:2932] T 5518112449d9448e98f5c779f1fa3248 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:05.316095 29873 catalog_manager.cc:5484] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index 6 to 8, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) evicted. New cstate: current_term: 2 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:05.320600 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 5518112449d9448e98f5c779f1fa3248 with delete_type TABLET_DATA_TOMBSTONED (TS d35c16dcb506426993811bb2519d7db3 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:45302
I20240501 14:10:05.321046 10983 tablet_replica.cc:330] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3: stopping tablet replica
I20240501 14:10:05.321357 10983 raft_consensus.cc:2229] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:05.321535 10983 raft_consensus.cc:2258] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:05.322224 10983 ts_tablet_manager.cc:1875] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:05.324123 10983 ts_tablet_manager.cc:1888] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240501 14:10:05.324191 10983 log.cc:1196] T 5518112449d9448e98f5c779f1fa3248 P d35c16dcb506426993811bb2519d7db3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/wals/5518112449d9448e98f5c779f1fa3248
I20240501 14:10:05.324712 29860 catalog_manager.cc:4830] TS d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): tablet 5518112449d9448e98f5c779f1fa3248 (table rebalance_test_table_5 [id=440878c63b0f4706b34fd42316991270]) successfully deleted
I20240501 14:10:05.493825 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:05.495398 32401 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:05.501310 32533 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:05.507519 29993 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:05.508428 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:05.521147 30386 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:05.525053 32665 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:05.531057 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:05.613415 30407 consensus_queue.cc:237] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:05.614867 32552 raft_consensus.cc:1261] T 8bb9687045d44efa888a69c7acde1105 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 2 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:10:05.615834 348 raft_consensus.cc:1261] T 8bb9687045d44efa888a69c7acde1105 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:10:05.616294 10880 consensus_queue.cc:1035] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:05.616912 10640 consensus_queue.cc:1035] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:10:05.618968 10640 raft_consensus.cc:2932] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:05.621657 348 raft_consensus.cc:2932] T 8bb9687045d44efa888a69c7acde1105 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:05.621811 32552 raft_consensus.cc:2932] T 8bb9687045d44efa888a69c7acde1105 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:05.624608 29870 catalog_manager.cc:5484] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index 2 to 4, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New cstate: current_term: 2 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:05.629299 32665 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bb9687045d44efa888a69c7acde1105 with delete_type TABLET_DATA_TOMBSTONED (TS db935501e9104de5b15c8780681d6c38 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35078
I20240501 14:10:05.633539 11083 tablet_replica.cc:330] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:10:05.633772 11083 raft_consensus.cc:2229] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:05.633951 11083 raft_consensus.cc:2258] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:05.635673 11083 ts_tablet_manager.cc:1875] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:05.638521 11083 ts_tablet_manager.cc:1888] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 14:10:05.639338 11083 log.cc:1196] T 8bb9687045d44efa888a69c7acde1105 P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/8bb9687045d44efa888a69c7acde1105
I20240501 14:10:05.639988 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet 8bb9687045d44efa888a69c7acde1105 (table rebalancer_extra_table_34 [id=c0df7ced31ed4653bf4f71f2bb576ed9]) successfully deleted
I20240501 14:10:05.693346 32683 consensus_queue.cc:237] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:05.694782 32552 raft_consensus.cc:1261] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:05.694900 32416 raft_consensus.cc:1261] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:10:05.695084 32620 consensus_peers.cc:485] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 -> Peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): Couldn't send request to peer 1d0bb69611af4876a2b3b33c4766e13a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5025e07cbfd543e3b4722c6c69e4a6b3. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:05.695230 9244 consensus_queue.cc:1035] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:05.695557 9244 consensus_queue.cc:1035] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:05.697569 9244 raft_consensus.cc:2932] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:10:05.698098 32416 raft_consensus.cc:2932] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:10:05.699787 32552 raft_consensus.cc:2932] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:10:05.701799 29870 catalog_manager.cc:5484] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New cstate: current_term: 1 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:10:05.723490 30014 consensus_queue.cc:237] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:05.725106 30406 raft_consensus.cc:1261] T deee2ab875274d4eaeab84e698241ca8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:05.725114 32552 raft_consensus.cc:1261] T deee2ab875274d4eaeab84e698241ca8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 14:10:05.725471 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: deee2ab875274d4eaeab84e698241ca8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:05.725646 11145 consensus_queue.cc:1035] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:05.725814 11063 consensus_queue.cc:1035] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:05.728644 11063 raft_consensus.cc:2932] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:10:05.730610 30406 raft_consensus.cc:2932] T deee2ab875274d4eaeab84e698241ca8 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:10:05.730618 32552 raft_consensus.cc:2932] T deee2ab875274d4eaeab84e698241ca8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:10:05.736749 29873 catalog_manager.cc:5484] T deee2ab875274d4eaeab84e698241ca8 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: config changed from index 5 to 6, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:10:05.754426 30276 consensus_queue.cc:237] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:10:05.755753 349 raft_consensus.cc:1261] T 36bf7d2992034055bccfd8146449b696 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:05.755836 32683 raft_consensus.cc:1261] T 36bf7d2992034055bccfd8146449b696 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:10:05.756021 30213 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36bf7d2992034055bccfd8146449b696. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:05.756191 10959 consensus_queue.cc:1035] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:05.756399 10564 consensus_queue.cc:1035] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:05.758179 10959 raft_consensus.cc:2932] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:10:05.758605 32683 raft_consensus.cc:2932] T 36bf7d2992034055bccfd8146449b696 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:10:05.761014 349 raft_consensus.cc:2932] T 36bf7d2992034055bccfd8146449b696 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:10:05.761816 29864 catalog_manager.cc:5484] T 36bf7d2992034055bccfd8146449b696 P db935501e9104de5b15c8780681d6c38 reported cstate change: config changed from index -1 to 2, VOTER 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) added. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:10:05.782887 30276 consensus_queue.cc:237] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:05.784746 32683 raft_consensus.cc:1261] T 75b67ccb9a334b25ae71b553a2a84a02 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 14:10:05.784847 30212 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): Couldn't send request to peer a91029b90bf9415a89a2f70736759ad3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75b67ccb9a334b25ae71b553a2a84a02. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:05.784853 30406 raft_consensus.cc:1261] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:10:05.785266 10564 consensus_queue.cc:1035] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:05.785434 11166 consensus_queue.cc:1035] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:10:05.787344 11166 raft_consensus.cc:2932] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:05.787739 30406 raft_consensus.cc:2932] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:05.792022 29873 catalog_manager.cc:5484] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: config changed from index 5 to 6, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:05.796689 32683 raft_consensus.cc:2932] T 75b67ccb9a334b25ae71b553a2a84a02 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:05.836365 32416 consensus_queue.cc:237] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:05.837684 30276 raft_consensus.cc:1261] T d6776209de5e405b982f863faa556727 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:05.837910 32552 raft_consensus.cc:1261] T d6776209de5e405b982f863faa556727 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:05.838328 11201 consensus_queue.cc:1035] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:10:05.838513 32358 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): Couldn't send request to peer 93f2518fbadf4f9991cda93afacb363e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6776209de5e405b982f863faa556727. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:05.838640 10979 consensus_queue.cc:1035] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:05.840473 11201 raft_consensus.cc:2932] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:05.840863 30276 raft_consensus.cc:2932] T d6776209de5e405b982f863faa556727 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:05.843048 32552 raft_consensus.cc:2932] T d6776209de5e405b982f863faa556727 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:05.845983 29868 catalog_manager.cc:5484] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:05.868260 349 consensus_queue.cc:237] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:05.869864 32684 raft_consensus.cc:1261] T caebf782fa60459b9e298aa9f1074449 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:05.869921 32552 raft_consensus.cc:1261] T caebf782fa60459b9e298aa9f1074449 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:10:05.870142 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Couldn't send request to peer d35c16dcb506426993811bb2519d7db3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: caebf782fa60459b9e298aa9f1074449. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:05.870380 10620 consensus_queue.cc:1035] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:05.870533 11221 consensus_queue.cc:1035] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:05.872249 10620 raft_consensus.cc:2932] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:05.872684 32684 raft_consensus.cc:2932] T caebf782fa60459b9e298aa9f1074449 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:05.874671 32552 raft_consensus.cc:2932] T caebf782fa60459b9e298aa9f1074449 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:05.875741 29864 catalog_manager.cc:5484] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:05.894412 30406 consensus_queue.cc:237] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:05.895790 32552 raft_consensus.cc:1261] T 27564c531f6147aeac4809082ffe2402 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:05.896045 30014 raft_consensus.cc:1261] T 27564c531f6147aeac4809082ffe2402 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:05.896224 11240 consensus_queue.cc:1035] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:05.896446 10640 consensus_queue.cc:1035] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:10:05.896649 30343 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): Couldn't send request to peer a91029b90bf9415a89a2f70736759ad3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27564c531f6147aeac4809082ffe2402. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:05.898141 10640 raft_consensus.cc:2932] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:05.898591 30014 raft_consensus.cc:2932] T 27564c531f6147aeac4809082ffe2402 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:05.900120 32552 raft_consensus.cc:2932] T 27564c531f6147aeac4809082ffe2402 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:05.901751 29868 catalog_manager.cc:5484] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: config changed from index -1 to 2, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New cstate: current_term: 1 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:05.923503 32415 consensus_queue.cc:237] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:05.924997 30276 raft_consensus.cc:1261] T ce04f87c84f54f1aaabb7cf10afe5b50 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:05.924976 30014 raft_consensus.cc:1261] T ce04f87c84f54f1aaabb7cf10afe5b50 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:10:05.925004 32356 consensus_peers.cc:485] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 -> Peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): Couldn't send request to peer d1c217469ae4478ebd49e4ed9f7352d7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce04f87c84f54f1aaabb7cf10afe5b50. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:05.925381 10979 consensus_queue.cc:1035] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:05.925565 11204 consensus_queue.cc:1035] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:05.927424 10979 raft_consensus.cc:2932] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:05.927829 30014 raft_consensus.cc:2932] T ce04f87c84f54f1aaabb7cf10afe5b50 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:05.930968 29864 catalog_manager.cc:5484] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:05.931376 30276 raft_consensus.cc:2932] T ce04f87c84f54f1aaabb7cf10afe5b50 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:05.950165 30014 consensus_queue.cc:237] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:05.951699 32415 raft_consensus.cc:1261] T 89d3e17f11a4460fa0d0a9a0b8342a66 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:05.951805 30277 raft_consensus.cc:1261] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:10:05.951911 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89d3e17f11a4460fa0d0a9a0b8342a66. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:05.952142 10724 consensus_queue.cc:1035] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:05.952379 11145 consensus_queue.cc:1035] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:05.954280 11145 raft_consensus.cc:2932] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:10:05.954682 32415 raft_consensus.cc:2932] T 89d3e17f11a4460fa0d0a9a0b8342a66 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:10:05.957542 30277 raft_consensus.cc:2932] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:10:05.958688 29873 catalog_manager.cc:5484] T 89d3e17f11a4460fa0d0a9a0b8342a66 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index -1 to 2, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) added. New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:10:05.976356 30013 consensus_queue.cc:237] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
W20240501 14:10:05.977885 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Couldn't send request to peer d35c16dcb506426993811bb2519d7db3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb828a6a5cd240f5acb591e7eae5e1a8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:05.980407 32415 raft_consensus.cc:1261] T eb828a6a5cd240f5acb591e7eae5e1a8 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:05.980746 10724 consensus_queue.cc:1035] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:05.981149 32552 raft_consensus.cc:1261] T eb828a6a5cd240f5acb591e7eae5e1a8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:05.981559 10724 consensus_queue.cc:1035] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:05.984197 10724 raft_consensus.cc:2932] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:05.985989 32552 raft_consensus.cc:2932] T eb828a6a5cd240f5acb591e7eae5e1a8 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:05.986780 32415 raft_consensus.cc:2932] T eb828a6a5cd240f5acb591e7eae5e1a8 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:05.987368 29864 catalog_manager.cc:5484] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:06.011436 30277 consensus_queue.cc:237] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:06.012746 30014 raft_consensus.cc:1261] T e7ae4cac38e0454d8b6e019208553dac P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:06.012746 32415 raft_consensus.cc:1261] T e7ae4cac38e0454d8b6e019208553dac P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:10:06.013185 30210 consensus_peers.cc:485] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a -> Peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): Couldn't send request to peer d1c217469ae4478ebd49e4ed9f7352d7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7ae4cac38e0454d8b6e019208553dac. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:06.013314 10959 consensus_queue.cc:1035] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:06.013447 9823 consensus_queue.cc:1035] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:06.016273 9823 raft_consensus.cc:2932] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:06.018117 32416 raft_consensus.cc:2932] T e7ae4cac38e0454d8b6e019208553dac P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:06.021818 30014 raft_consensus.cc:2932] T e7ae4cac38e0454d8b6e019208553dac P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:06.021941 29873 catalog_manager.cc:5484] T e7ae4cac38e0454d8b6e019208553dac P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:06.040840 32415 consensus_queue.cc:237] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:06.042053 30406 raft_consensus.cc:1261] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:06.042131 30276 raft_consensus.cc:1261] T 42636717a44e4e3782037c48d8913307 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:06.042445 10979 consensus_queue.cc:1035] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:10:06.042646 32358 consensus_peers.cc:485] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 -> Peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): Couldn't send request to peer 93f2518fbadf4f9991cda93afacb363e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42636717a44e4e3782037c48d8913307. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:06.042753 10979 consensus_queue.cc:1035] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:06.044796 10979 raft_consensus.cc:2932] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:06.045223 30276 raft_consensus.cc:2932] T 42636717a44e4e3782037c48d8913307 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:06.046787 30406 raft_consensus.cc:2932] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:06.048208 29873 catalog_manager.cc:5484] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:06.068409 349 consensus_queue.cc:237] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:06.069916 32684 raft_consensus.cc:1261] T ea69585d641342b8a1340339bb2732b3 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:06.069916 32554 raft_consensus.cc:1261] T ea69585d641342b8a1340339bb2732b3 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:06.070358 10620 consensus_queue.cc:1035] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:06.070533 11064 consensus_queue.cc:1035] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:10:06.070714 32752 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): Couldn't send request to peer 1d0bb69611af4876a2b3b33c4766e13a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea69585d641342b8a1340339bb2732b3. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:06.072592 11221 raft_consensus.cc:2932] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:10:06.074609 32684 raft_consensus.cc:2932] T ea69585d641342b8a1340339bb2732b3 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:10:06.074680 32554 raft_consensus.cc:2932] T ea69585d641342b8a1340339bb2732b3 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:10:06.077809 29868 catalog_manager.cc:5484] T ea69585d641342b8a1340339bb2732b3 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: config changed from index -1 to 2, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) added. New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:10:06.107914 32552 consensus_queue.cc:237] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:06.109402 32415 raft_consensus.cc:1261] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:06.109546 30406 raft_consensus.cc:1261] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 5f8a0cfdfdca4148a75686dc88059cf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:10:06.109702 32490 consensus_peers.cc:485] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 -> Peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Couldn't send request to peer d35c16dcb506426993811bb2519d7db3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89ce3093e24347f3a4e121af452c6361. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:06.109942 11371 consensus_queue.cc:1035] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:06.110163 10619 consensus_queue.cc:1035] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:06.115824 10619 raft_consensus.cc:2932] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:06.116281 32415 raft_consensus.cc:2932] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:06.119817 29870 catalog_manager.cc:5484] T 89ce3093e24347f3a4e121af452c6361 P 5f8a0cfdfdca4148a75686dc88059cf3 reported cstate change: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New cstate: current_term: 1 leader_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:06.121153 30406 raft_consensus.cc:2932] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:06.204412 11416 ts_tablet_manager.cc:902] T 36bf7d2992034055bccfd8146449b696 P 5f8a0cfdfdca4148a75686dc88059cf3: Initiating tablet copy from peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
F20240501 14:10:06.204635 11416 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0517ff3519644b489b912f8e5a36274c.queue_time_us, tablet-download-pool-0517ff3519644b489b912f8e5a36274c.queue_time_us) (tablet-download-pool-0517ff3519644b489b912f8e5a36274c.run_cpu_time_us, tablet-download-pool-0517ff3519644b489b912f8e5a36274c.run_cpu_time_us) (tablet-download-pool-0517ff3519644b489b912f8e5a36274c.run_wall_time_us, tablet-download-pool-0517ff3519644b489b912f8e5a36274c.run_wall_time_us) (tablet-download-pool-19f203c753ef481dafc091ea2d60f56f.queue_time_us, tablet-download-pool-19f203c753ef481dafc091ea2d60f56f.queue_time_us) (tablet-download-pool-19f203c753ef481dafc091ea2d60f56f.run_cpu_time_us, tablet-download-pool-19f203c753ef481dafc091ea2d60f56f.run_cpu_time_us) (tablet-download-pool-19f203c753ef481dafc091ea2d60f56f.run_wall_time_us, tablet-download-pool-19f203c753ef481dafc091ea2d60f56f.run_wall_time_us) (tablet-download-pool-205a0fda164d46a29a031070f1d0c369.queue_time_us, tablet-download-pool-205a0fda164d46a29a031070f1d0c369.queue_time_us) (tablet-download-pool-205a0fda164d46a29a031070f1d0c369.run_cpu_time_us, tablet-download-pool-205a0fda164d46a29a031070f1d0c369.run_cpu_time_us) (tablet-download-pool-205a0fda164d46a29a031070f1d0c369.run_wall_time_us, tablet-download-pool-205a0fda164d46a29a031070f1d0c369.run_wall_time_us) (tablet-download-pool-36bf7d2992034055bccfd8146449b696.queue_time_us, tablet-download-pool-36bf7d2992034055bccfd8146449b696.queue_time_us) (tablet-download-pool-44949ddf1eac45cfa4fc8e1c532cf7bb.queue_time_us, tablet-download-pool-44949ddf1eac45cfa4fc8e1c532cf7bb.queue_time_us) (tablet-download-pool-44949ddf1eac45cfa4fc8e1c532cf7bb.run_cpu_time_us, tablet-download-pool-44949ddf1eac45cfa4fc8e1c532cf7bb.run_cpu_time_us) (tablet-download-pool-44949ddf1eac45cfa4fc8e1c532cf7bb.run_wall_time_us, tablet-download-pool-44949ddf1eac45cfa4fc8e1c532cf7bb.run_wall_time_us) (tablet-download-pool-5518112449d9448e98f5c779f1fa3248.queue_time_us, tablet-download-pool-5518112449d9448e98f5c779f1fa3248.queue_time_us) (tablet-download-pool-5518112449d9448e98f5c779f1fa3248.run_cpu_time_us, tablet-download-pool-5518112449d9448e98f5c779f1fa3248.run_cpu_time_us) (tablet-download-pool-5518112449d9448e98f5c779f1fa3248.run_wall_time_us, tablet-download-pool-5518112449d9448e98f5c779f1fa3248.run_wall_time_us) (tablet-download-pool-5da2beba68ba414281025fcee91ca0d0.queue_time_us, tablet-download-pool-5da2beba68ba414281025fcee91ca0d0.queue_time_us) (tablet-download-pool-5da2beba68ba414281025fcee91ca0d0.run_cpu_time_us, tablet-download-pool-5da2beba68ba414281025fcee91ca0d0.run_cpu_time_us) (tablet-download-pool-5da2beba68ba414281025fcee91ca0d0.run_wall_time_us, tablet-download-pool-5da2beba68ba414281025fcee91ca0d0.run_wall_time_us) (tablet-download-pool-8d5e1afe21654080b5aa3eb877e8f1d9.queue_time_us, tablet-download-pool-8d5e1afe21654080b5aa3eb877e8f1d9.queue_time_us) (tablet-download-pool-8d5e1afe21654080b5aa3eb877e8f1d9.run_cpu_time_us, tablet-download-pool-8d5e1afe21654080b5aa3eb877e8f1d9.run_cpu_time_us) (tablet-download-pool-8d5e1afe21654080b5aa3eb877e8f1d9.run_wall_time_us, tablet-download-pool-8d5e1afe21654080b5aa3eb877e8f1d9.run_wall_time_us) (tablet-download-pool-bf0712a5e11c46a291b90bfb491b8b25.queue_time_us, tablet-download-pool-bf0712a5e11c46a291b90bfb491b8b25.queue_time_us) (tablet-download-pool-bf0712a5e11c46a291b90bfb491b8b25.run_cpu_time_us, tablet-download-pool-bf0712a5e11c46a291b90bfb491b8b25.run_cpu_time_us) (tablet-download-pool-bf0712a5e11c46a291b90bfb491b8b25.run_wall_time_us, tablet-download-pool-bf0712a5e11c46a291b90bfb491b8b25.run_wall_time_us) (tablet-download-pool-c362bd01a59b4bcf85eeb59713e58b64.queue_time_us, tablet-download-pool-c362bd01a59b4bcf85eeb59713e58b64.queue_time_us) (tablet-download-pool-c362bd01a59b4bcf85eeb59713e58b64.run_cpu_time_us, tablet-download-pool-c362bd01a59b4bcf85eeb59713e58b64.run_cpu_time_us) (tablet-download-pool-c362bd01a59b4bcf85eeb59713e58b64.run_wall_time_us, tablet-download-pool-c362bd01a59b4bcf85eeb59713e58b64.run_wall_time_us) (tablet-download-pool-c5d406c64b2e432a88e8c675b91dd144.queue_time_us, tablet-download-pool-c5d406c64b2e432a88e8c675b91dd144.queue_time_us) (tablet-download-pool-c5d406c64b2e432a88e8c675b91dd144.run_cpu_time_us, tablet-download-pool-c5d406c64b2e432a88e8c675b91dd144.run_cpu_time_us) (tablet-download-pool-c5d406c64b2e432a88e8c675b91dd144.run_wall_time_us, tablet-download-pool-c5d406c64b2e432a88e8c675b91dd144.run_wall_time_us) (tablet-download-pool-c7df6b8d76714eebbeacc85f403d3405.queue_time_us, tablet-download-pool-c7df6b8d76714eebbeacc85f403d3405.queue_time_us) (tablet-download-pool-c7df6b8d76714eebbeacc85f403d3405.run_cpu_time_us, tablet-download-pool-c7df6b8d76714eebbeacc85f403d3405.run_cpu_time_us) (tablet-download-pool-c7df6b8d76714eebbeacc85f403d3405.run_wall_time_us, tablet-download-pool-c7df6b8d76714eebbeacc85f403d3405.run_wall_time_us) (tablet-download-pool-cfc701c19cd74d0db2f90ce580230177.queue_time_us, tablet-download-pool-cfc701c19cd74d0db2f90ce580230177.queue_time_us) (tablet-download-pool-cfc701c19cd74d0db2f90ce580230177.run_cpu_time_us, tablet-download-pool-cfc701c19cd74d0db2f90ce580230177.run_cpu_time_us) (tablet-download-pool-cfc701c19cd74d0db2f90ce580230177.run_wall_time_us, tablet-download-pool-cfc701c19cd74d0db2f90ce580230177.run_wall_time_us) (tablet-download-pool-deee2ab875274d4eaeab84e698241ca8.queue_time_us, tablet-download-pool-deee2ab875274d4eaeab84e698241ca8.queue_time_us) (tablet-download-pool-deee2ab875274d4eaeab84e698241ca8.run_cpu_time_us, tablet-download-pool-deee2ab875274d4eaeab84e698241ca8.run_cpu_time_us) (tablet-download-pool-deee2ab875274d4eaeab84e698241ca8.run_wall_time_us, tablet-download-pool-deee2ab875274d4eaeab84e698241ca8.run_wall_time_us) (tablet-download-pool-f2bc06e76e2a41429d5625ebf99e5e5c.queue_time_us, tablet-download-pool-f2bc06e76e2a41429d5625ebf99e5e5c.queue_time_us) (tablet-download-pool-f2bc06e76e2a41429d5625ebf99e5e5c.run_cpu_time_us, tablet-download-pool-f2bc06e76e2a41429d5625ebf99e5e5c.run_cpu_time_us) (tablet-download-pool-f2bc06e76e2a41429d5625ebf99e5e5c.run_wall_time_us, tablet-download-pool-f2bc06e76e2a41429d5625ebf99e5e5c.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fc57cb9ddcd google::LogMessage::Fail() at ??:0
@ 0x7fc57cba1b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fc57cb9d7cc google::LogMessage::Flush() at ??:0
@ 0x7fc57cb9ef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 14:10:06.224619 11418 ts_tablet_manager.cc:902] T d6776209de5e405b982f863faa556727 P 93f2518fbadf4f9991cda93afacb363e: Initiating tablet copy from peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
@ 0x7fc57e007044 kudu::TraceMetrics::InternName() at ??:0
I20240501 14:10:06.225281 11418 tablet_copy_client.cc:308] T d6776209de5e405b982f863faa556727 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.133:46265
I20240501 14:10:06.225872 32443 tablet_copy_service.cc:140] P a91029b90bf9415a89a2f70736759ad3: Received BeginTabletCopySession request for tablet d6776209de5e405b982f863faa556727 from peer 93f2518fbadf4f9991cda93afacb363e ({username='slave'} at 127.22.252.129:56321)
I20240501 14:10:06.225991 32443 tablet_copy_service.cc:161] P a91029b90bf9415a89a2f70736759ad3: Beginning new tablet copy session on tablet d6776209de5e405b982f863faa556727 from peer 93f2518fbadf4f9991cda93afacb363e at {username='slave'} at 127.22.252.129:56321: session id = 93f2518fbadf4f9991cda93afacb363e-d6776209de5e405b982f863faa556727
I20240501 14:10:06.227334 32443 tablet_copy_source_session.cc:205] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x7fc57dfebf98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fc57dfe9efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fc580b9c77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fc580ba5a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 14:10:06.247278 11418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6776209de5e405b982f863faa556727. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fc580c2676f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fc580c3a4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 14:10:06.258049 11423 ts_tablet_manager.cc:902] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a: Initiating tablet copy from peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:10:06.258710 11423 tablet_copy_client.cc:308] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.135:41915
I20240501 14:10:06.259277 11418 tablet_copy_client.cc:783] T d6776209de5e405b982f863faa556727 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:06.259317 32705 tablet_copy_service.cc:140] P db935501e9104de5b15c8780681d6c38: Received BeginTabletCopySession request for tablet 5025e07cbfd543e3b4722c6c69e4a6b3 from peer 1d0bb69611af4876a2b3b33c4766e13a ({username='slave'} at 127.22.252.131:49203)
I20240501 14:10:06.259428 32705 tablet_copy_service.cc:161] P db935501e9104de5b15c8780681d6c38: Beginning new tablet copy session on tablet 5025e07cbfd543e3b4722c6c69e4a6b3 from peer 1d0bb69611af4876a2b3b33c4766e13a at {username='slave'} at 127.22.252.131:49203: session id = 1d0bb69611af4876a2b3b33c4766e13a-5025e07cbfd543e3b4722c6c69e4a6b3
I20240501 14:10:06.260325 32705 tablet_copy_source_session.cc:205] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x7fc580c25226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240501 14:10:06.263749 11418 tablet_copy_client.cc:647] T d6776209de5e405b982f863faa556727 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Starting download of 1 WAL segments...
@ 0x7fc580c34c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55b25bbd475c std::function<>::operator()() at ??:0
I20240501 14:10:06.265115 11423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5025e07cbfd543e3b4722c6c69e4a6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:06.265112 11418 tablet_copy_client.cc:523] T d6776209de5e405b982f863faa556727 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:06.266894 11418 tablet_bootstrap.cc:492] T d6776209de5e405b982f863faa556727 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
@ 0x7fc57dfeee06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc57dfef6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc57dff103b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55b25bbd475c std::function<>::operator()() at ??:0
@ 0x7fc57dfde807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc57e81c6db start_thread at ??:0
@ 0x7fc57bd5471f clone at ??:0
I20240501 14:10:06.291489 11423 tablet_copy_client.cc:783] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:06.291782 11423 tablet_copy_client.cc:647] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:06.295127 11423 tablet_copy_client.cc:523] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:06.298626 11423 tablet_bootstrap.cc:492] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
I20240501 14:10:06.299711 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:06.304136 11444 ts_tablet_manager.cc:902] T 75b67ccb9a334b25ae71b553a2a84a02 P a91029b90bf9415a89a2f70736759ad3: Initiating tablet copy from peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:06.305606 11444 tablet_copy_client.cc:308] T 75b67ccb9a334b25ae71b553a2a84a02 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.131:43103
I20240501 14:10:06.306392 30297 tablet_copy_service.cc:140] P 1d0bb69611af4876a2b3b33c4766e13a: Received BeginTabletCopySession request for tablet 75b67ccb9a334b25ae71b553a2a84a02 from peer a91029b90bf9415a89a2f70736759ad3 ({username='slave'} at 127.22.252.133:60853)
I20240501 14:10:06.306479 30297 tablet_copy_service.cc:161] P 1d0bb69611af4876a2b3b33c4766e13a: Beginning new tablet copy session on tablet 75b67ccb9a334b25ae71b553a2a84a02 from peer a91029b90bf9415a89a2f70736759ad3 at {username='slave'} at 127.22.252.133:60853: session id = a91029b90bf9415a89a2f70736759ad3-75b67ccb9a334b25ae71b553a2a84a02
I20240501 14:10:06.306507 11418 tablet_bootstrap.cc:492] T d6776209de5e405b982f863faa556727 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:06.306841 11418 tablet_bootstrap.cc:492] T d6776209de5e405b982f863faa556727 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap complete.
I20240501 14:10:06.307029 11418 ts_tablet_manager.cc:1367] T d6776209de5e405b982f863faa556727 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.003s
I20240501 14:10:06.307546 30297 tablet_copy_source_session.cc:205] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:06.307644 11418 raft_consensus.cc:348] T d6776209de5e405b982f863faa556727 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:06.307780 11418 raft_consensus.cc:729] T d6776209de5e405b982f863faa556727 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:10:06.307916 11418 consensus_queue.cc:260] T d6776209de5e405b982f863faa556727 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:06.308372 11418 ts_tablet_manager.cc:1398] T d6776209de5e405b982f863faa556727 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 14:10:06.310380 11444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b67ccb9a334b25ae71b553a2a84a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:06.310436 32443 tablet_copy_service.cc:342] P a91029b90bf9415a89a2f70736759ad3: Request end of tablet copy session 93f2518fbadf4f9991cda93afacb363e-d6776209de5e405b982f863faa556727 received from {username='slave'} at 127.22.252.129:56321
I20240501 14:10:06.310504 32443 tablet_copy_service.cc:433] P a91029b90bf9415a89a2f70736759ad3: ending tablet copy session 93f2518fbadf4f9991cda93afacb363e-d6776209de5e405b982f863faa556727 on tablet d6776209de5e405b982f863faa556727 with peer 93f2518fbadf4f9991cda93afacb363e
I20240501 14:10:06.313632 11444 tablet_copy_client.cc:783] T 75b67ccb9a334b25ae71b553a2a84a02 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:06.313771 11444 tablet_copy_client.cc:647] T 75b67ccb9a334b25ae71b553a2a84a02 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:06.314616 29993 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:06.317162 30386 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:06.324651 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:06.329146 11444 tablet_copy_client.cc:523] T 75b67ccb9a334b25ae71b553a2a84a02 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:06.334367 32665 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:06.340395 11444 tablet_bootstrap.cc:492] T 75b67ccb9a334b25ae71b553a2a84a02 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:10:06.348721 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:06.349035 32401 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:06.365372 11423 tablet_bootstrap.cc:492] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:06.365665 11423 tablet_bootstrap.cc:492] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap complete.
I20240501 14:10:06.365845 11423 ts_tablet_manager.cc:1367] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.000s
I20240501 14:10:06.366313 11423 raft_consensus.cc:348] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:06.366452 11423 raft_consensus.cc:729] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:10:06.366585 11423 consensus_queue.cc:260] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:06.366923 11423 ts_tablet_manager.cc:1398] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:06.367341 32705 tablet_copy_service.cc:342] P db935501e9104de5b15c8780681d6c38: Request end of tablet copy session 1d0bb69611af4876a2b3b33c4766e13a-5025e07cbfd543e3b4722c6c69e4a6b3 received from {username='slave'} at 127.22.252.131:49203
I20240501 14:10:06.367420 32705 tablet_copy_service.cc:433] P db935501e9104de5b15c8780681d6c38: ending tablet copy session 1d0bb69611af4876a2b3b33c4766e13a-5025e07cbfd543e3b4722c6c69e4a6b3 on tablet 5025e07cbfd543e3b4722c6c69e4a6b3 with peer 1d0bb69611af4876a2b3b33c4766e13a
I20240501 14:10:06.374773 11450 ts_tablet_manager.cc:902] T deee2ab875274d4eaeab84e698241ca8 P db935501e9104de5b15c8780681d6c38: Initiating tablet copy from peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:10:06.393203 11450 tablet_copy_client.cc:308] T deee2ab875274d4eaeab84e698241ca8 P db935501e9104de5b15c8780681d6c38: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.129:41387
I20240501 14:10:06.393738 30035 tablet_copy_service.cc:140] P 93f2518fbadf4f9991cda93afacb363e: Received BeginTabletCopySession request for tablet deee2ab875274d4eaeab84e698241ca8 from peer db935501e9104de5b15c8780681d6c38 ({username='slave'} at 127.22.252.135:45653)
I20240501 14:10:06.393844 30035 tablet_copy_service.cc:161] P 93f2518fbadf4f9991cda93afacb363e: Beginning new tablet copy session on tablet deee2ab875274d4eaeab84e698241ca8 from peer db935501e9104de5b15c8780681d6c38 at {username='slave'} at 127.22.252.135:45653: session id = db935501e9104de5b15c8780681d6c38-deee2ab875274d4eaeab84e698241ca8
I20240501 14:10:06.394924 30035 tablet_copy_source_session.cc:205] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:06.396253 11450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deee2ab875274d4eaeab84e698241ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:06.408316 11450 tablet_copy_client.cc:783] T deee2ab875274d4eaeab84e698241ca8 P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:06.408457 11456 ts_tablet_manager.cc:902] T e7ae4cac38e0454d8b6e019208553dac P d1c217469ae4478ebd49e4ed9f7352d7: Initiating tablet copy from peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:06.408545 11450 tablet_copy_client.cc:647] T deee2ab875274d4eaeab84e698241ca8 P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:06.409039 11456 tablet_copy_client.cc:308] T e7ae4cac38e0454d8b6e019208553dac P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.131:43103
I20240501 14:10:06.409740 30297 tablet_copy_service.cc:140] P 1d0bb69611af4876a2b3b33c4766e13a: Received BeginTabletCopySession request for tablet e7ae4cac38e0454d8b6e019208553dac from peer d1c217469ae4478ebd49e4ed9f7352d7 ({username='slave'} at 127.22.252.132:39047)
I20240501 14:10:06.409847 30297 tablet_copy_service.cc:161] P 1d0bb69611af4876a2b3b33c4766e13a: Beginning new tablet copy session on tablet e7ae4cac38e0454d8b6e019208553dac from peer d1c217469ae4478ebd49e4ed9f7352d7 at {username='slave'} at 127.22.252.132:39047: session id = d1c217469ae4478ebd49e4ed9f7352d7-e7ae4cac38e0454d8b6e019208553dac
I20240501 14:10:06.410913 30297 tablet_copy_source_session.cc:205] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:06.411828 11463 ts_tablet_manager.cc:902] T 27564c531f6147aeac4809082ffe2402 P a91029b90bf9415a89a2f70736759ad3: Initiating tablet copy from peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:10:06.412437 11463 tablet_copy_client.cc:308] T 27564c531f6147aeac4809082ffe2402 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.132:34357
I20240501 14:10:06.413236 30427 tablet_copy_service.cc:140] P d1c217469ae4478ebd49e4ed9f7352d7: Received BeginTabletCopySession request for tablet 27564c531f6147aeac4809082ffe2402 from peer a91029b90bf9415a89a2f70736759ad3 ({username='slave'} at 127.22.252.133:43857)
I20240501 14:10:06.413322 30427 tablet_copy_service.cc:161] P d1c217469ae4478ebd49e4ed9f7352d7: Beginning new tablet copy session on tablet 27564c531f6147aeac4809082ffe2402 from peer a91029b90bf9415a89a2f70736759ad3 at {username='slave'} at 127.22.252.133:43857: session id = a91029b90bf9415a89a2f70736759ad3-27564c531f6147aeac4809082ffe2402
I20240501 14:10:06.414290 30427 tablet_copy_source_session.cc:205] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:06.414242 11450 tablet_copy_client.cc:523] T deee2ab875274d4eaeab84e698241ca8 P db935501e9104de5b15c8780681d6c38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:06.416527 11450 tablet_bootstrap.cc:492] T deee2ab875274d4eaeab84e698241ca8 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:10:06.416188 11456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7ae4cac38e0454d8b6e019208553dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:06.422155 11463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27564c531f6147aeac4809082ffe2402. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:06.426154 11456 tablet_copy_client.cc:783] T e7ae4cac38e0454d8b6e019208553dac P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:06.426319 11456 tablet_copy_client.cc:647] T e7ae4cac38e0454d8b6e019208553dac P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:06.427600 11463 tablet_copy_client.cc:783] T 27564c531f6147aeac4809082ffe2402 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:06.427752 11463 tablet_copy_client.cc:647] T 27564c531f6147aeac4809082ffe2402 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:06.429873 11463 tablet_copy_client.cc:523] T 27564c531f6147aeac4809082ffe2402 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:06.431295 11463 tablet_bootstrap.cc:492] T 27564c531f6147aeac4809082ffe2402 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:10:06.432849 11456 tablet_copy_client.cc:523] T e7ae4cac38e0454d8b6e019208553dac P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:06.434173 11456 tablet_bootstrap.cc:492] T e7ae4cac38e0454d8b6e019208553dac P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:10:06.437978 11456 tablet_bootstrap.cc:492] T e7ae4cac38e0454d8b6e019208553dac P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:06.438228 11456 tablet_bootstrap.cc:492] T e7ae4cac38e0454d8b6e019208553dac P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap complete.
I20240501 14:10:06.438477 11456 ts_tablet_manager.cc:1367] T e7ae4cac38e0454d8b6e019208553dac P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.005s
I20240501 14:10:06.439059 11456 raft_consensus.cc:348] T e7ae4cac38e0454d8b6e019208553dac P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:06.439179 11456 raft_consensus.cc:729] T e7ae4cac38e0454d8b6e019208553dac P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:10:06.439314 11456 consensus_queue.cc:260] T e7ae4cac38e0454d8b6e019208553dac P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:06.441432 11456 ts_tablet_manager.cc:1398] T e7ae4cac38e0454d8b6e019208553dac P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:10:06.442281 30297 tablet_copy_service.cc:342] P 1d0bb69611af4876a2b3b33c4766e13a: Request end of tablet copy session d1c217469ae4478ebd49e4ed9f7352d7-e7ae4cac38e0454d8b6e019208553dac received from {username='slave'} at 127.22.252.132:39047
I20240501 14:10:06.442356 30297 tablet_copy_service.cc:433] P 1d0bb69611af4876a2b3b33c4766e13a: ending tablet copy session d1c217469ae4478ebd49e4ed9f7352d7-e7ae4cac38e0454d8b6e019208553dac on tablet e7ae4cac38e0454d8b6e019208553dac with peer d1c217469ae4478ebd49e4ed9f7352d7
I20240501 14:10:06.443548 11444 tablet_bootstrap.cc:492] T 75b67ccb9a334b25ae71b553a2a84a02 P a91029b90bf9415a89a2f70736759ad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:06.446254 11444 tablet_bootstrap.cc:492] T 75b67ccb9a334b25ae71b553a2a84a02 P a91029b90bf9415a89a2f70736759ad3: Bootstrap complete.
I20240501 14:10:06.446442 11444 ts_tablet_manager.cc:1367] T 75b67ccb9a334b25ae71b553a2a84a02 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.106s user 0.032s sys 0.005s
I20240501 14:10:06.446933 11444 raft_consensus.cc:348] T 75b67ccb9a334b25ae71b553a2a84a02 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:06.447057 11444 raft_consensus.cc:729] T 75b67ccb9a334b25ae71b553a2a84a02 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:10:06.447227 11444 consensus_queue.cc:260] T 75b67ccb9a334b25ae71b553a2a84a02 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:06.447758 11444 ts_tablet_manager.cc:1398] T 75b67ccb9a334b25ae71b553a2a84a02 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:06.448379 11477 ts_tablet_manager.cc:902] T eb828a6a5cd240f5acb591e7eae5e1a8 P d35c16dcb506426993811bb2519d7db3: Initiating tablet copy from peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:10:06.448873 30297 tablet_copy_service.cc:342] P 1d0bb69611af4876a2b3b33c4766e13a: Request end of tablet copy session a91029b90bf9415a89a2f70736759ad3-75b67ccb9a334b25ae71b553a2a84a02 received from {username='slave'} at 127.22.252.133:60853
I20240501 14:10:06.448940 30297 tablet_copy_service.cc:433] P 1d0bb69611af4876a2b3b33c4766e13a: ending tablet copy session a91029b90bf9415a89a2f70736759ad3-75b67ccb9a334b25ae71b553a2a84a02 on tablet 75b67ccb9a334b25ae71b553a2a84a02 with peer a91029b90bf9415a89a2f70736759ad3
I20240501 14:10:06.450717 11477 tablet_copy_client.cc:308] T eb828a6a5cd240f5acb591e7eae5e1a8 P d35c16dcb506426993811bb2519d7db3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.129:41387
I20240501 14:10:06.451300 30035 tablet_copy_service.cc:140] P 93f2518fbadf4f9991cda93afacb363e: Received BeginTabletCopySession request for tablet eb828a6a5cd240f5acb591e7eae5e1a8 from peer d35c16dcb506426993811bb2519d7db3 ({username='slave'} at 127.22.252.130:37241)
I20240501 14:10:06.451392 30035 tablet_copy_service.cc:161] P 93f2518fbadf4f9991cda93afacb363e: Beginning new tablet copy session on tablet eb828a6a5cd240f5acb591e7eae5e1a8 from peer d35c16dcb506426993811bb2519d7db3 at {username='slave'} at 127.22.252.130:37241: session id = d35c16dcb506426993811bb2519d7db3-eb828a6a5cd240f5acb591e7eae5e1a8
I20240501 14:10:06.452454 30035 tablet_copy_source_session.cc:205] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:06.452905 11456 ts_tablet_manager.cc:902] T ce04f87c84f54f1aaabb7cf10afe5b50 P d1c217469ae4478ebd49e4ed9f7352d7: Initiating tablet copy from peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:10:06.453385 11477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb828a6a5cd240f5acb591e7eae5e1a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:06.455921 11477 tablet_copy_client.cc:783] T eb828a6a5cd240f5acb591e7eae5e1a8 P d35c16dcb506426993811bb2519d7db3: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:06.456111 11477 tablet_copy_client.cc:647] T eb828a6a5cd240f5acb591e7eae5e1a8 P d35c16dcb506426993811bb2519d7db3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:06.457166 11477 tablet_copy_client.cc:523] T eb828a6a5cd240f5acb591e7eae5e1a8 P d35c16dcb506426993811bb2519d7db3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:06.458600 11477 tablet_bootstrap.cc:492] T eb828a6a5cd240f5acb591e7eae5e1a8 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:10:06.459951 11456 tablet_copy_client.cc:308] T ce04f87c84f54f1aaabb7cf10afe5b50 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.133:46265
I20240501 14:10:06.460438 11463 tablet_bootstrap.cc:492] T 27564c531f6147aeac4809082ffe2402 P a91029b90bf9415a89a2f70736759ad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:06.460693 11463 tablet_bootstrap.cc:492] T 27564c531f6147aeac4809082ffe2402 P a91029b90bf9415a89a2f70736759ad3: Bootstrap complete.
I20240501 14:10:06.460851 11463 ts_tablet_manager.cc:1367] T 27564c531f6147aeac4809082ffe2402 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.001s
I20240501 14:10:06.461309 11463 raft_consensus.cc:348] T 27564c531f6147aeac4809082ffe2402 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:06.461416 11463 raft_consensus.cc:729] T 27564c531f6147aeac4809082ffe2402 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:10:06.461540 11463 consensus_queue.cc:260] T 27564c531f6147aeac4809082ffe2402 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:06.461827 11463 ts_tablet_manager.cc:1398] T 27564c531f6147aeac4809082ffe2402 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:06.462007 32443 tablet_copy_service.cc:140] P a91029b90bf9415a89a2f70736759ad3: Received BeginTabletCopySession request for tablet ce04f87c84f54f1aaabb7cf10afe5b50 from peer d1c217469ae4478ebd49e4ed9f7352d7 ({username='slave'} at 127.22.252.132:53277)
I20240501 14:10:06.462080 32443 tablet_copy_service.cc:161] P a91029b90bf9415a89a2f70736759ad3: Beginning new tablet copy session on tablet ce04f87c84f54f1aaabb7cf10afe5b50 from peer d1c217469ae4478ebd49e4ed9f7352d7 at {username='slave'} at 127.22.252.132:53277: session id = d1c217469ae4478ebd49e4ed9f7352d7-ce04f87c84f54f1aaabb7cf10afe5b50
I20240501 14:10:06.462108 11477 tablet_bootstrap.cc:492] T eb828a6a5cd240f5acb591e7eae5e1a8 P d35c16dcb506426993811bb2519d7db3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:06.462333 11477 tablet_bootstrap.cc:492] T eb828a6a5cd240f5acb591e7eae5e1a8 P d35c16dcb506426993811bb2519d7db3: Bootstrap complete.
I20240501 14:10:06.462463 11477 ts_tablet_manager.cc:1367] T eb828a6a5cd240f5acb591e7eae5e1a8 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:10:06.463016 32443 tablet_copy_source_session.cc:205] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:06.464394 30427 tablet_copy_service.cc:342] P d1c217469ae4478ebd49e4ed9f7352d7: Request end of tablet copy session a91029b90bf9415a89a2f70736759ad3-27564c531f6147aeac4809082ffe2402 received from {username='slave'} at 127.22.252.133:43857
I20240501 14:10:06.464469 30427 tablet_copy_service.cc:433] P d1c217469ae4478ebd49e4ed9f7352d7: ending tablet copy session a91029b90bf9415a89a2f70736759ad3-27564c531f6147aeac4809082ffe2402 on tablet 27564c531f6147aeac4809082ffe2402 with peer a91029b90bf9415a89a2f70736759ad3
I20240501 14:10:06.464972 11456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce04f87c84f54f1aaabb7cf10afe5b50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:06.467113 11456 tablet_copy_client.cc:783] T ce04f87c84f54f1aaabb7cf10afe5b50 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:06.467235 11456 tablet_copy_client.cc:647] T ce04f87c84f54f1aaabb7cf10afe5b50 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:06.470350 11477 raft_consensus.cc:348] T eb828a6a5cd240f5acb591e7eae5e1a8 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:06.470458 11477 raft_consensus.cc:729] T eb828a6a5cd240f5acb591e7eae5e1a8 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:10:06.470569 11477 consensus_queue.cc:260] T eb828a6a5cd240f5acb591e7eae5e1a8 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:06.471604 11477 ts_tablet_manager.cc:1398] T eb828a6a5cd240f5acb591e7eae5e1a8 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240501 14:10:06.472368 30035 tablet_copy_service.cc:342] P 93f2518fbadf4f9991cda93afacb363e: Request end of tablet copy session d35c16dcb506426993811bb2519d7db3-eb828a6a5cd240f5acb591e7eae5e1a8 received from {username='slave'} at 127.22.252.130:37241
I20240501 14:10:06.472437 30035 tablet_copy_service.cc:433] P 93f2518fbadf4f9991cda93afacb363e: ending tablet copy session d35c16dcb506426993811bb2519d7db3-eb828a6a5cd240f5acb591e7eae5e1a8 on tablet eb828a6a5cd240f5acb591e7eae5e1a8 with peer d35c16dcb506426993811bb2519d7db3
I20240501 14:10:06.473435 11456 tablet_copy_client.cc:523] T ce04f87c84f54f1aaabb7cf10afe5b50 P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:06.474651 11456 tablet_bootstrap.cc:492] T ce04f87c84f54f1aaabb7cf10afe5b50 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:10:06.478629 11477 ts_tablet_manager.cc:902] T caebf782fa60459b9e298aa9f1074449 P d35c16dcb506426993811bb2519d7db3: Initiating tablet copy from peer 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:10:06.478899 11450 tablet_bootstrap.cc:492] T deee2ab875274d4eaeab84e698241ca8 P db935501e9104de5b15c8780681d6c38: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:06.479192 11450 tablet_bootstrap.cc:492] T deee2ab875274d4eaeab84e698241ca8 P db935501e9104de5b15c8780681d6c38: Bootstrap complete.
I20240501 14:10:06.479386 11450 ts_tablet_manager.cc:1367] T deee2ab875274d4eaeab84e698241ca8 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.063s user 0.029s sys 0.000s
I20240501 14:10:06.479730 11477 tablet_copy_client.cc:308] T caebf782fa60459b9e298aa9f1074449 P d35c16dcb506426993811bb2519d7db3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.136:45937
I20240501 14:10:06.479821 11450 raft_consensus.cc:348] T deee2ab875274d4eaeab84e698241ca8 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:06.479934 11450 raft_consensus.cc:729] T deee2ab875274d4eaeab84e698241ca8 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:10:06.480029 11450 consensus_queue.cc:260] T deee2ab875274d4eaeab84e698241ca8 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:06.480093 369 tablet_copy_service.cc:140] P 9ecd907cf5634ac98f3f54f3dbec6849: Received BeginTabletCopySession request for tablet caebf782fa60459b9e298aa9f1074449 from peer d35c16dcb506426993811bb2519d7db3 ({username='slave'} at 127.22.252.130:43657)
I20240501 14:10:06.480173 369 tablet_copy_service.cc:161] P 9ecd907cf5634ac98f3f54f3dbec6849: Beginning new tablet copy session on tablet caebf782fa60459b9e298aa9f1074449 from peer d35c16dcb506426993811bb2519d7db3 at {username='slave'} at 127.22.252.130:43657: session id = d35c16dcb506426993811bb2519d7db3-caebf782fa60459b9e298aa9f1074449
I20240501 14:10:06.480329 11450 ts_tablet_manager.cc:1398] T deee2ab875274d4eaeab84e698241ca8 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 14:10:06.481042 369 tablet_copy_source_session.cc:205] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:06.481917 11477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caebf782fa60459b9e298aa9f1074449. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:06.483974 11477 tablet_copy_client.cc:783] T caebf782fa60459b9e298aa9f1074449 P d35c16dcb506426993811bb2519d7db3: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:06.484149 11477 tablet_copy_client.cc:647] T caebf782fa60459b9e298aa9f1074449 P d35c16dcb506426993811bb2519d7db3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:06.484851 11456 tablet_bootstrap.cc:492] T ce04f87c84f54f1aaabb7cf10afe5b50 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:06.485098 11456 tablet_bootstrap.cc:492] T ce04f87c84f54f1aaabb7cf10afe5b50 P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap complete.
I20240501 14:10:06.485370 11456 ts_tablet_manager.cc:1367] T ce04f87c84f54f1aaabb7cf10afe5b50 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240501 14:10:06.486259 11456 raft_consensus.cc:348] T ce04f87c84f54f1aaabb7cf10afe5b50 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:06.486451 11456 raft_consensus.cc:729] T ce04f87c84f54f1aaabb7cf10afe5b50 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:10:06.486606 11456 consensus_queue.cc:260] T ce04f87c84f54f1aaabb7cf10afe5b50 P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:06.486898 11456 ts_tablet_manager.cc:1398] T ce04f87c84f54f1aaabb7cf10afe5b50 P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:06.492015 32443 tablet_copy_service.cc:342] P a91029b90bf9415a89a2f70736759ad3: Request end of tablet copy session d1c217469ae4478ebd49e4ed9f7352d7-ce04f87c84f54f1aaabb7cf10afe5b50 received from {username='slave'} at 127.22.252.132:53277
I20240501 14:10:06.492075 32443 tablet_copy_service.cc:433] P a91029b90bf9415a89a2f70736759ad3: ending tablet copy session d1c217469ae4478ebd49e4ed9f7352d7-ce04f87c84f54f1aaabb7cf10afe5b50 on tablet ce04f87c84f54f1aaabb7cf10afe5b50 with peer d1c217469ae4478ebd49e4ed9f7352d7
I20240501 14:10:06.492791 30035 tablet_copy_service.cc:342] P 93f2518fbadf4f9991cda93afacb363e: Request end of tablet copy session db935501e9104de5b15c8780681d6c38-deee2ab875274d4eaeab84e698241ca8 received from {username='slave'} at 127.22.252.135:45653
I20240501 14:10:06.492872 30035 tablet_copy_service.cc:433] P 93f2518fbadf4f9991cda93afacb363e: ending tablet copy session db935501e9104de5b15c8780681d6c38-deee2ab875274d4eaeab84e698241ca8 on tablet deee2ab875274d4eaeab84e698241ca8 with peer db935501e9104de5b15c8780681d6c38
I20240501 14:10:06.498486 11477 tablet_copy_client.cc:523] T caebf782fa60459b9e298aa9f1074449 P d35c16dcb506426993811bb2519d7db3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:06.500170 11477 tablet_bootstrap.cc:492] T caebf782fa60459b9e298aa9f1074449 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:10:06.506150 11477 tablet_bootstrap.cc:492] T caebf782fa60459b9e298aa9f1074449 P d35c16dcb506426993811bb2519d7db3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:06.506414 11477 tablet_bootstrap.cc:492] T caebf782fa60459b9e298aa9f1074449 P d35c16dcb506426993811bb2519d7db3: Bootstrap complete.
I20240501 14:10:06.506570 11477 ts_tablet_manager.cc:1367] T caebf782fa60459b9e298aa9f1074449 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20240501 14:10:06.507148 11477 raft_consensus.cc:348] T caebf782fa60459b9e298aa9f1074449 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:06.507279 11477 raft_consensus.cc:729] T caebf782fa60459b9e298aa9f1074449 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:10:06.507468 11477 consensus_queue.cc:260] T caebf782fa60459b9e298aa9f1074449 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:06.509564 11477 ts_tablet_manager.cc:1398] T caebf782fa60459b9e298aa9f1074449 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:06.510475 369 tablet_copy_service.cc:342] P 9ecd907cf5634ac98f3f54f3dbec6849: Request end of tablet copy session d35c16dcb506426993811bb2519d7db3-caebf782fa60459b9e298aa9f1074449 received from {username='slave'} at 127.22.252.130:43657
I20240501 14:10:06.510555 369 tablet_copy_service.cc:433] P 9ecd907cf5634ac98f3f54f3dbec6849: ending tablet copy session d35c16dcb506426993811bb2519d7db3-caebf782fa60459b9e298aa9f1074449 on tablet caebf782fa60459b9e298aa9f1074449 with peer d35c16dcb506426993811bb2519d7db3
I20240501 14:10:06.528719 11450 ts_tablet_manager.cc:902] T 89d3e17f11a4460fa0d0a9a0b8342a66 P db935501e9104de5b15c8780681d6c38: Initiating tablet copy from peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:10:06.529565 11450 tablet_copy_client.cc:308] T 89d3e17f11a4460fa0d0a9a0b8342a66 P db935501e9104de5b15c8780681d6c38: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.129:41387
I20240501 14:10:06.530048 30035 tablet_copy_service.cc:140] P 93f2518fbadf4f9991cda93afacb363e: Received BeginTabletCopySession request for tablet 89d3e17f11a4460fa0d0a9a0b8342a66 from peer db935501e9104de5b15c8780681d6c38 ({username='slave'} at 127.22.252.135:45653)
I20240501 14:10:06.530160 30035 tablet_copy_service.cc:161] P 93f2518fbadf4f9991cda93afacb363e: Beginning new tablet copy session on tablet 89d3e17f11a4460fa0d0a9a0b8342a66 from peer db935501e9104de5b15c8780681d6c38 at {username='slave'} at 127.22.252.135:45653: session id = db935501e9104de5b15c8780681d6c38-89d3e17f11a4460fa0d0a9a0b8342a66
I20240501 14:10:06.530907 30035 tablet_copy_source_session.cc:205] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:06.531773 11450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89d3e17f11a4460fa0d0a9a0b8342a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:06.534397 11450 tablet_copy_client.cc:783] T 89d3e17f11a4460fa0d0a9a0b8342a66 P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:06.534559 11450 tablet_copy_client.cc:647] T 89d3e17f11a4460fa0d0a9a0b8342a66 P db935501e9104de5b15c8780681d6c38: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:06.534629 11418 ts_tablet_manager.cc:902] T 42636717a44e4e3782037c48d8913307 P 93f2518fbadf4f9991cda93afacb363e: Initiating tablet copy from peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:10:06.535095 11418 tablet_copy_client.cc:308] T 42636717a44e4e3782037c48d8913307 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.133:46265
I20240501 14:10:06.535555 32443 tablet_copy_service.cc:140] P a91029b90bf9415a89a2f70736759ad3: Received BeginTabletCopySession request for tablet 42636717a44e4e3782037c48d8913307 from peer 93f2518fbadf4f9991cda93afacb363e ({username='slave'} at 127.22.252.129:56321)
I20240501 14:10:06.535667 32443 tablet_copy_service.cc:161] P a91029b90bf9415a89a2f70736759ad3: Beginning new tablet copy session on tablet 42636717a44e4e3782037c48d8913307 from peer 93f2518fbadf4f9991cda93afacb363e at {username='slave'} at 127.22.252.129:56321: session id = 93f2518fbadf4f9991cda93afacb363e-42636717a44e4e3782037c48d8913307
I20240501 14:10:06.536412 11450 tablet_copy_client.cc:523] T 89d3e17f11a4460fa0d0a9a0b8342a66 P db935501e9104de5b15c8780681d6c38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:06.536744 32443 tablet_copy_source_session.cc:205] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:06.537710 11450 tablet_bootstrap.cc:492] T 89d3e17f11a4460fa0d0a9a0b8342a66 P db935501e9104de5b15c8780681d6c38: Bootstrap starting.
I20240501 14:10:06.540518 11418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42636717a44e4e3782037c48d8913307. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:06.541445 11450 tablet_bootstrap.cc:492] T 89d3e17f11a4460fa0d0a9a0b8342a66 P db935501e9104de5b15c8780681d6c38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:06.541698 11450 tablet_bootstrap.cc:492] T 89d3e17f11a4460fa0d0a9a0b8342a66 P db935501e9104de5b15c8780681d6c38: Bootstrap complete.
I20240501 14:10:06.541853 11450 ts_tablet_manager.cc:1367] T 89d3e17f11a4460fa0d0a9a0b8342a66 P db935501e9104de5b15c8780681d6c38: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:10:06.542296 11450 raft_consensus.cc:348] T 89d3e17f11a4460fa0d0a9a0b8342a66 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:06.542419 11450 raft_consensus.cc:729] T 89d3e17f11a4460fa0d0a9a0b8342a66 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db935501e9104de5b15c8780681d6c38, State: Initialized, Role: FOLLOWER
I20240501 14:10:06.542474 11418 tablet_copy_client.cc:783] T 42636717a44e4e3782037c48d8913307 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:06.542542 11450 consensus_queue.cc:260] T 89d3e17f11a4460fa0d0a9a0b8342a66 P db935501e9104de5b15c8780681d6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:06.542610 11418 tablet_copy_client.cc:647] T 42636717a44e4e3782037c48d8913307 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:06.542832 11450 ts_tablet_manager.cc:1398] T 89d3e17f11a4460fa0d0a9a0b8342a66 P db935501e9104de5b15c8780681d6c38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:06.543247 30035 tablet_copy_service.cc:342] P 93f2518fbadf4f9991cda93afacb363e: Request end of tablet copy session db935501e9104de5b15c8780681d6c38-89d3e17f11a4460fa0d0a9a0b8342a66 received from {username='slave'} at 127.22.252.135:45653
I20240501 14:10:06.543311 30035 tablet_copy_service.cc:433] P 93f2518fbadf4f9991cda93afacb363e: ending tablet copy session db935501e9104de5b15c8780681d6c38-89d3e17f11a4460fa0d0a9a0b8342a66 on tablet 89d3e17f11a4460fa0d0a9a0b8342a66 with peer db935501e9104de5b15c8780681d6c38
I20240501 14:10:06.545325 11418 tablet_copy_client.cc:523] T 42636717a44e4e3782037c48d8913307 P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:06.546885 11418 tablet_bootstrap.cc:492] T 42636717a44e4e3782037c48d8913307 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:10:06.550096 11418 tablet_bootstrap.cc:492] T 42636717a44e4e3782037c48d8913307 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:06.550307 11418 tablet_bootstrap.cc:492] T 42636717a44e4e3782037c48d8913307 P 93f2518fbadf4f9991cda93afacb363e: Bootstrap complete.
I20240501 14:10:06.550495 11418 ts_tablet_manager.cc:1367] T 42636717a44e4e3782037c48d8913307 P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:10:06.550910 11418 raft_consensus.cc:348] T 42636717a44e4e3782037c48d8913307 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:06.551038 11418 raft_consensus.cc:729] T 42636717a44e4e3782037c48d8913307 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:10:06.551165 11418 consensus_queue.cc:260] T 42636717a44e4e3782037c48d8913307 P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:06.551442 11418 ts_tablet_manager.cc:1398] T 42636717a44e4e3782037c48d8913307 P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:06.552477 32443 tablet_copy_service.cc:342] P a91029b90bf9415a89a2f70736759ad3: Request end of tablet copy session 93f2518fbadf4f9991cda93afacb363e-42636717a44e4e3782037c48d8913307 received from {username='slave'} at 127.22.252.129:56321
I20240501 14:10:06.552553 32443 tablet_copy_service.cc:433] P a91029b90bf9415a89a2f70736759ad3: ending tablet copy session 93f2518fbadf4f9991cda93afacb363e-42636717a44e4e3782037c48d8913307 on tablet 42636717a44e4e3782037c48d8913307 with peer 93f2518fbadf4f9991cda93afacb363e
W20240501 14:10:06.577730 30210 connection.cc:535] server connection from 127.22.252.134:55213 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:06.577737 30213 connection.cc:535] client connection to 127.22.252.134:33497 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:06.577908 30213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:06.578743 30082 connection.cc:535] client connection to 127.22.252.134:33497 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:06.578768 32621 connection.cc:535] client connection to 127.22.252.134:33497 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:06.578876 30082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:06.578889 32621 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:06.579313 32753 connection.cc:535] client connection to 127.22.252.134:33497 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:06.579315 32359 connection.cc:535] client connection to 127.22.252.134:33497 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:06.579401 32753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:06.579420 32359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:06.579981 30344 connection.cc:535] client connection to 127.22.252.134:33497 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:06.580086 30344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:06.579857 29951 connection.cc:535] client connection to 127.22.252.134:33497 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:06.581030 29951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:06.583721 11518 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:10:06.579795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.252.134:33497 (local address 127.22.252.130:48677)
0501 14:10:06.581351 (+ 1556us) negotiation.cc:107] Waiting for socket to connect
0501 14:10:06.581364 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 14:10:06.581445 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:10:06.583228 (+ 1783us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1488,"thread_start_us":56,"threads_started":1}
W20240501 14:10:06.583721 11513 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:10:06.578664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.252.134:33497 (local address 127.22.252.131:44917)
0501 14:10:06.578967 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0501 14:10:06.578983 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 14:10:06.579079 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:10:06.583261 (+ 4182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":56,"threads_started":1}
W20240501 14:10:06.583904 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.584152 11523 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:10:06.581746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.252.134:33497 (local address 127.22.252.129:37803)
0501 14:10:06.583715 (+ 1969us) negotiation.cc:107] Waiting for socket to connect
0501 14:10:06.583779 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":1895,"thread_start_us":70,"threads_started":1}
W20240501 14:10:06.584363 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.584378 11520 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:10:06.580565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.252.134:33497 (local address 127.22.252.136:59969)
0501 14:10:06.580916 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0501 14:10:06.580931 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 14:10:06.581090 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:10:06.583946 (+ 2856us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":109,"threads_started":1}
W20240501 14:10:06.584507 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.584585 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.584633 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.584676 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.585464 11522 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:10:06.581465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.252.134:33497 (local address 127.22.252.133:50297)
0501 14:10:06.585037 (+ 3572us) negotiation.cc:107] Waiting for socket to connect
0501 14:10:06.585117 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":3499,"thread_start_us":70,"threads_started":1}
W20240501 14:10:06.585551 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.585604 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.585636 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.585665 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.585665 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:06.585703 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.585722 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.585736 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.585768 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.586122 11524 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:10:06.581864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.252.134:33497 (local address 127.22.252.135:45107)
0501 14:10:06.582270 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0501 14:10:06.582282 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 14:10:06.582377 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:10:06.584403 (+ 2026us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":56,"threads_started":1}
W20240501 14:10:06.586282 32621 consensus_peers.cc:485] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.586342 32621 consensus_peers.cc:485] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.586378 32621 consensus_peers.cc:485] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.586467 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.586524 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.586573 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.586617 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.586658 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.586689 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.586813 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.587296 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.598178 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:06.617740 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.617916 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.624748 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.629478 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.633695 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.635497 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.635633 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.662007 32621 consensus_peers.cc:485] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:06.665329 11423 ts_tablet_manager.cc:902] T ea69585d641342b8a1340339bb2732b3 P 1d0bb69611af4876a2b3b33c4766e13a: Initiating tablet copy from peer 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:10:06.665843 11423 tablet_copy_client.cc:308] T ea69585d641342b8a1340339bb2732b3 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.136:45937
I20240501 14:10:06.666323 369 tablet_copy_service.cc:140] P 9ecd907cf5634ac98f3f54f3dbec6849: Received BeginTabletCopySession request for tablet ea69585d641342b8a1340339bb2732b3 from peer 1d0bb69611af4876a2b3b33c4766e13a ({username='slave'} at 127.22.252.131:38303)
I20240501 14:10:06.666419 369 tablet_copy_service.cc:161] P 9ecd907cf5634ac98f3f54f3dbec6849: Beginning new tablet copy session on tablet ea69585d641342b8a1340339bb2732b3 from peer 1d0bb69611af4876a2b3b33c4766e13a at {username='slave'} at 127.22.252.131:38303: session id = 1d0bb69611af4876a2b3b33c4766e13a-ea69585d641342b8a1340339bb2732b3
I20240501 14:10:06.667425 369 tablet_copy_source_session.cc:205] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849: Tablet Copy: opened 0 blocks and 1 log segments
W20240501 14:10:06.667538 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:06.668700 11423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea69585d641342b8a1340339bb2732b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:06.670742 11423 tablet_copy_client.cc:783] T ea69585d641342b8a1340339bb2732b3 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:06.670852 11423 tablet_copy_client.cc:647] T ea69585d641342b8a1340339bb2732b3 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Starting download of 1 WAL segments...
W20240501 14:10:06.671590 32621 consensus_peers.cc:485] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:06.676399 11423 tablet_copy_client.cc:523] T ea69585d641342b8a1340339bb2732b3 P 1d0bb69611af4876a2b3b33c4766e13a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:06.677647 11423 tablet_bootstrap.cc:492] T ea69585d641342b8a1340339bb2732b3 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap starting.
W20240501 14:10:06.679687 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:06.681674 11423 tablet_bootstrap.cc:492] T ea69585d641342b8a1340339bb2732b3 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:06.681890 11423 tablet_bootstrap.cc:492] T ea69585d641342b8a1340339bb2732b3 P 1d0bb69611af4876a2b3b33c4766e13a: Bootstrap complete.
I20240501 14:10:06.682049 11423 ts_tablet_manager.cc:1367] T ea69585d641342b8a1340339bb2732b3 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:10:06.682493 11423 raft_consensus.cc:348] T ea69585d641342b8a1340339bb2732b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:06.682675 11423 raft_consensus.cc:729] T ea69585d641342b8a1340339bb2732b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Initialized, Role: FOLLOWER
I20240501 14:10:06.682847 11423 consensus_queue.cc:260] T ea69585d641342b8a1340339bb2732b3 P 1d0bb69611af4876a2b3b33c4766e13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:06.684098 11423 ts_tablet_manager.cc:1398] T ea69585d641342b8a1340339bb2732b3 P 1d0bb69611af4876a2b3b33c4766e13a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:06.684554 369 tablet_copy_service.cc:342] P 9ecd907cf5634ac98f3f54f3dbec6849: Request end of tablet copy session 1d0bb69611af4876a2b3b33c4766e13a-ea69585d641342b8a1340339bb2732b3 received from {username='slave'} at 127.22.252.131:38303
I20240501 14:10:06.684629 369 tablet_copy_service.cc:433] P 9ecd907cf5634ac98f3f54f3dbec6849: ending tablet copy session 1d0bb69611af4876a2b3b33c4766e13a-ea69585d641342b8a1340339bb2732b3 on tablet ea69585d641342b8a1340339bb2732b3 with peer 1d0bb69611af4876a2b3b33c4766e13a
W20240501 14:10:06.689031 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.689102 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:06.705353 32621 consensus_peers.cc:485] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:06.729447 30014 raft_consensus.cc:1203] T d6776209de5e405b982f863faa556727 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:10:06.746362 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:06.746644 30277 raft_consensus.cc:1203] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:10:06.751192 30386 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:06.753825 32665 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:06.755664 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:06.755676 32401 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:06.759694 29993 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:06.761478 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:06.811529 32416 raft_consensus.cc:1203] T 75b67ccb9a334b25ae71b553a2a84a02 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
W20240501 14:10:06.829380 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:06.843672 32684 raft_consensus.cc:1203] T deee2ab875274d4eaeab84e698241ca8 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 14:10:06.849354 30145 raft_consensus.cc:1203] T eb828a6a5cd240f5acb591e7eae5e1a8 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:10:06.859107 30407 raft_consensus.cc:1203] T e7ae4cac38e0454d8b6e019208553dac P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:10:06.880156 32416 raft_consensus.cc:1203] T 27564c531f6147aeac4809082ffe2402 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:10:06.918128 30407 raft_consensus.cc:1203] T ce04f87c84f54f1aaabb7cf10afe5b50 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:10:06.922241 32684 raft_consensus.cc:1203] T 89d3e17f11a4460fa0d0a9a0b8342a66 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:10:06.946331 30386 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:06.947327 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:06.948267 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:06.955734 29993 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:06.964560 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:06.966851 32665 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:06.969568 32401 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:06.977360 30145 raft_consensus.cc:1203] T caebf782fa60459b9e298aa9f1074449 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:10:07.005582 30014 raft_consensus.cc:1203] T 42636717a44e4e3782037c48d8913307 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 14:10:07.009598 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.151888 29993 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:07.158102 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:07.165676 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:07.168704 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:07.171331 32665 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:07.188032 32401 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:07.194533 30386 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:07.252118 30277 consensus_queue.cc:237] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:07.253237 30407 raft_consensus.cc:1261] T e7ae4cac38e0454d8b6e019208553dac P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:07.253270 30013 raft_consensus.cc:1261] T e7ae4cac38e0454d8b6e019208553dac P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:07.253674 10959 consensus_queue.cc:1035] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:07.253829 11711 consensus_queue.cc:1035] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:07.256368 10959 raft_consensus.cc:2932] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:07.258461 30407 raft_consensus.cc:2932] T e7ae4cac38e0454d8b6e019208553dac P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:07.261477 30013 raft_consensus.cc:2932] T e7ae4cac38e0454d8b6e019208553dac P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:07.261974 29871 catalog_manager.cc:5484] T e7ae4cac38e0454d8b6e019208553dac P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:07.265398 32401 tablet_service.cc:1514] Processing DeleteTablet for tablet e7ae4cac38e0454d8b6e019208553dac with delete_type TABLET_DATA_TOMBSTONED (TS a91029b90bf9415a89a2f70736759ad3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53090
I20240501 14:10:07.265700 11724 tablet_replica.cc:330] T e7ae4cac38e0454d8b6e019208553dac P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:10:07.265872 11724 raft_consensus.cc:2229] T e7ae4cac38e0454d8b6e019208553dac P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:07.266045 11724 raft_consensus.cc:2258] T e7ae4cac38e0454d8b6e019208553dac P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:07.266713 11724 ts_tablet_manager.cc:1875] T e7ae4cac38e0454d8b6e019208553dac P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:07.268947 11724 ts_tablet_manager.cc:1888] T e7ae4cac38e0454d8b6e019208553dac P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:10:07.269017 11724 log.cc:1196] T e7ae4cac38e0454d8b6e019208553dac P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/e7ae4cac38e0454d8b6e019208553dac
I20240501 14:10:07.269526 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet e7ae4cac38e0454d8b6e019208553dac (table rebalancer_extra_table_44 [id=67fbf296b4324bdb855fc05097cf2d0f]) successfully deleted
I20240501 14:10:07.273394 10417 raft_consensus.cc:482] T bd36b13ca2274fc19b9d32d202e94f83 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.273530 10417 raft_consensus.cc:504] T bd36b13ca2274fc19b9d32d202e94f83 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:07.274008 10417 leader_election.cc:288] T bd36b13ca2274fc19b9d32d202e94f83 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:10:07.274354 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd36b13ca2274fc19b9d32d202e94f83" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
W20240501 14:10:07.275277 30082 leader_election.cc:334] T bd36b13ca2274fc19b9d32d202e94f83 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.275341 30082 leader_election.cc:302] T bd36b13ca2274fc19b9d32d202e94f83 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d35c16dcb506426993811bb2519d7db3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849
I20240501 14:10:07.275630 10417 raft_consensus.cc:2726] T bd36b13ca2274fc19b9d32d202e94f83 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:07.280870 30277 raft_consensus.cc:1203] T ea69585d641342b8a1340339bb2732b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:10:07.289108 11240 raft_consensus.cc:482] T 9cf197259d9648b5a742dcbb74e7f89b P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.289211 11240 raft_consensus.cc:504] T 9cf197259d9648b5a742dcbb74e7f89b P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:07.289588 11240 leader_election.cc:288] T 9cf197259d9648b5a742dcbb74e7f89b P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:07.289863 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf197259d9648b5a742dcbb74e7f89b" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
W20240501 14:10:07.290516 30344 leader_election.cc:334] T 9cf197259d9648b5a742dcbb74e7f89b P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.290585 30344 leader_election.cc:302] T 9cf197259d9648b5a742dcbb74e7f89b P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7; no voters: 1d0bb69611af4876a2b3b33c4766e13a, 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.290709 11240 raft_consensus.cc:2726] T 9cf197259d9648b5a742dcbb74e7f89b P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:07.300395 10798 raft_consensus.cc:482] T 8f4669cad44f4a3c9e01c4bba3c14734 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.300527 10798 raft_consensus.cc:504] T 8f4669cad44f4a3c9e01c4bba3c14734 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:07.300930 10798 leader_election.cc:288] T 8f4669cad44f4a3c9e01c4bba3c14734 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:10:07.301486 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4669cad44f4a3c9e01c4bba3c14734" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
W20240501 14:10:07.302273 32621 leader_election.cc:334] T 8f4669cad44f4a3c9e01c4bba3c14734 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.302413 32621 leader_election.cc:302] T 8f4669cad44f4a3c9e01c4bba3c14734 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db935501e9104de5b15c8780681d6c38; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3
I20240501 14:10:07.302639 10798 raft_consensus.cc:2726] T 8f4669cad44f4a3c9e01c4bba3c14734 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:07.359907 10978 raft_consensus.cc:482] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.360044 10978 raft_consensus.cc:504] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:07.360452 10978 leader_election.cc:288] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:10:07.361349 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3110aca9b1d643e4b07d0a869015b6f7" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
W20240501 14:10:07.362036 32359 leader_election.cc:334] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.362107 32359 leader_election.cc:302] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, 93f2518fbadf4f9991cda93afacb363e
I20240501 14:10:07.362249 11204 raft_consensus.cc:2726] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:07.394045 30386 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:07.396742 30255 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:07.397751 32665 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:10:07.403707 30125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:07.408473 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:07.411144 32401 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:07.416597 29993 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:07.420598 11711 raft_consensus.cc:482] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.420760 11711 raft_consensus.cc:504] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:07.421204 11711 leader_election.cc:288] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:10:07.421838 11694 raft_consensus.cc:482] T a46319e2fdff439e969092ead5d7e11e P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.421942 11694 raft_consensus.cc:504] T a46319e2fdff439e969092ead5d7e11e P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:10:07.422330 11694 leader_election.cc:288] T a46319e2fdff439e969092ead5d7e11e P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:10:07.425247 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46319e2fdff439e969092ead5d7e11e" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
W20240501 14:10:07.425360 30213 leader_election.cc:334] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:07.427173 30082 leader_election.cc:334] T a46319e2fdff439e969092ead5d7e11e P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.427246 30082 leader_election.cc:302] T a46319e2fdff439e969092ead5d7e11e P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d35c16dcb506426993811bb2519d7db3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849
I20240501 14:10:07.429809 10417 raft_consensus.cc:2726] T a46319e2fdff439e969092ead5d7e11e P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:07.431520 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf197259d9648b5a742dcbb74e7f89b" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:10:07.431672 30407 raft_consensus.cc:2454] T 9cf197259d9648b5a742dcbb74e7f89b P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:10:07.431960 30210 leader_election.cc:302] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters: 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.432185 11711 raft_consensus.cc:2781] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:07.432265 11711 raft_consensus.cc:482] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.432315 11711 raft_consensus.cc:3037] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:07.433075 11064 raft_consensus.cc:482] T 7cb8f294e9704c549e1e0590b1429fe5 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.433171 11064 raft_consensus.cc:504] T 7cb8f294e9704c549e1e0590b1429fe5 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:07.433405 11711 raft_consensus.cc:504] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:07.433516 11064 leader_election.cc:288] T 7cb8f294e9704c549e1e0590b1429fe5 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:10:07.433758 11711 leader_election.cc:288] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
W20240501 14:10:07.435497 30213 leader_election.cc:334] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:07.436882 32753 leader_election.cc:334] T 7cb8f294e9704c549e1e0590b1429fe5 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.437080 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf197259d9648b5a742dcbb74e7f89b" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:10:07.437194 30407 raft_consensus.cc:3037] T 9cf197259d9648b5a742dcbb74e7f89b P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:07.438151 30407 raft_consensus.cc:2454] T 9cf197259d9648b5a742dcbb74e7f89b P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 2.
I20240501 14:10:07.439702 30210 leader_election.cc:302] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters: 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.439936 11711 raft_consensus.cc:2781] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:07.440016 11711 raft_consensus.cc:686] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [term 2 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:10:07.440140 11711 consensus_queue.cc:237] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:07.440605 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb8f294e9704c549e1e0590b1429fe5" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:10:07.441013 32753 leader_election.cc:302] T 7cb8f294e9704c549e1e0590b1429fe5 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3
I20240501 14:10:07.441633 11064 raft_consensus.cc:2726] T 7cb8f294e9704c549e1e0590b1429fe5 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:07.443040 29873 catalog_manager.cc:5484] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 1 to 2, leader changed from 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 2 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:10:07.464274 11204 raft_consensus.cc:482] T b816de53d6a74e798a3b4d8988165c54 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.464380 11204 raft_consensus.cc:504] T b816de53d6a74e798a3b4d8988165c54 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:07.464761 11204 leader_election.cc:288] T b816de53d6a74e798a3b4d8988165c54 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:10:07.465417 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b816de53d6a74e798a3b4d8988165c54" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:10:07.465566 10798 raft_consensus.cc:482] T aa8ba266a8d0406bb93d6c578fe6c7cb P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.465662 10798 raft_consensus.cc:504] T aa8ba266a8d0406bb93d6c578fe6c7cb P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:07.466064 10798 leader_election.cc:288] T aa8ba266a8d0406bb93d6c578fe6c7cb P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
W20240501 14:10:07.466168 32359 leader_election.cc:334] T b816de53d6a74e798a3b4d8988165c54 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.466240 32359 leader_election.cc:302] T b816de53d6a74e798a3b4d8988165c54 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3; no voters: 1d0bb69611af4876a2b3b33c4766e13a, 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.466413 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8ba266a8d0406bb93d6c578fe6c7cb" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:10:07.466826 11204 raft_consensus.cc:2726] T b816de53d6a74e798a3b4d8988165c54 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:07.467221 32621 leader_election.cc:334] T aa8ba266a8d0406bb93d6c578fe6c7cb P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.467291 32621 leader_election.cc:302] T aa8ba266a8d0406bb93d6c578fe6c7cb P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db935501e9104de5b15c8780681d6c38; no voters: 1d0bb69611af4876a2b3b33c4766e13a, 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.467422 10798 raft_consensus.cc:2726] T aa8ba266a8d0406bb93d6c578fe6c7cb P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:07.474973 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.478228 11063 raft_consensus.cc:482] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.478335 11063 raft_consensus.cc:504] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:07.478688 11063 leader_election.cc:288] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:10:07.479197 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3110aca9b1d643e4b07d0a869015b6f7" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:10:07.479346 32415 raft_consensus.cc:2454] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:10:07.479696 29950 leader_election.cc:302] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:10:07.479840 11063 raft_consensus.cc:2781] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:07.479898 11063 raft_consensus.cc:482] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.479941 11063 raft_consensus.cc:3037] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:10:07.480124 29951 leader_election.cc:334] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.480913 11063 raft_consensus.cc:504] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:07.481216 11063 leader_election.cc:288] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:10:07.481650 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3110aca9b1d643e4b07d0a869015b6f7" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:10:07.481750 32415 raft_consensus.cc:3037] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:07.482851 32415 raft_consensus.cc:2454] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 2.
W20240501 14:10:07.483119 29951 leader_election.cc:334] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.483546 29950 leader_election.cc:302] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, a91029b90bf9415a89a2f70736759ad3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.483809 11145 raft_consensus.cc:2781] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:07.483868 11145 raft_consensus.cc:686] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [term 2 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:10:07.483999 11145 consensus_queue.cc:237] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:07.486250 29873 catalog_manager.cc:5484] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 1 to 2, leader changed from 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 2 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:07.503825 11714 raft_consensus.cc:482] T f7264487a83e4574a7946e53f5dc8612 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.503959 11714 raft_consensus.cc:504] T f7264487a83e4574a7946e53f5dc8612 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:07.504447 11714 leader_election.cc:288] T f7264487a83e4574a7946e53f5dc8612 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:10:07.504832 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7264487a83e4574a7946e53f5dc8612" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
W20240501 14:10:07.505415 30213 leader_election.cc:334] T f7264487a83e4574a7946e53f5dc8612 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.505491 30213 leader_election.cc:302] T f7264487a83e4574a7946e53f5dc8612 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, d1c217469ae4478ebd49e4ed9f7352d7
I20240501 14:10:07.505623 11714 raft_consensus.cc:2726] T f7264487a83e4574a7946e53f5dc8612 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:07.528640 11711 raft_consensus.cc:482] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.528745 11711 raft_consensus.cc:504] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:07.529107 11711 leader_election.cc:288] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:10:07.530751 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21efaac3a3d643679c5abaaddb9e5a3a" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
W20240501 14:10:07.531575 30213 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.531649 30213 leader_election.cc:302] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:07.531845 11711 raft_consensus.cc:2726] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:07.537091 11240 raft_consensus.cc:482] T 22bb82c5d65640128fc3271212aafb60 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.537202 11240 raft_consensus.cc:504] T 22bb82c5d65640128fc3271212aafb60 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:07.537597 11240 leader_election.cc:288] T 22bb82c5d65640128fc3271212aafb60 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:10:07.538059 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22bb82c5d65640128fc3271212aafb60" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
W20240501 14:10:07.539450 30344 leader_election.cc:334] T 22bb82c5d65640128fc3271212aafb60 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.539530 30344 leader_election.cc:302] T 22bb82c5d65640128fc3271212aafb60 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, a91029b90bf9415a89a2f70736759ad3
I20240501 14:10:07.539702 11240 raft_consensus.cc:2726] T 22bb82c5d65640128fc3271212aafb60 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:07.555370 10798 raft_consensus.cc:482] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.555495 10798 raft_consensus.cc:504] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:07.555888 10798 leader_election.cc:288] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:07.556484 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e03e76474584273ae050595a2286559" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
W20240501 14:10:07.557189 32621 leader_election.cc:334] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.557271 32621 leader_election.cc:302] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db935501e9104de5b15c8780681d6c38; no voters: 1d0bb69611af4876a2b3b33c4766e13a, 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.557475 10798 raft_consensus.cc:2726] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:07.563115 11711 raft_consensus.cc:482] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.563230 11711 raft_consensus.cc:504] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:07.563634 11711 leader_election.cc:288] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
W20240501 14:10:07.564918 30213 leader_election.cc:334] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.565752 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e03e76474584273ae050595a2286559" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:10:07.565891 32684 raft_consensus.cc:2454] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:10:07.566170 30210 leader_election.cc:302] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, db935501e9104de5b15c8780681d6c38; no voters: 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.566380 11711 raft_consensus.cc:2781] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:07.566439 11711 raft_consensus.cc:482] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.566488 11711 raft_consensus.cc:3037] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:07.567528 11711 raft_consensus.cc:504] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:07.567859 11711 leader_election.cc:288] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:10:07.568447 11240 raft_consensus.cc:482] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.568565 11240 raft_consensus.cc:504] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:07.568928 11240 leader_election.cc:288] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
W20240501 14:10:07.569171 30213 leader_election.cc:334] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.569406 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7264487a83e4574a7946e53f5dc8612" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:10:07.569525 30277 raft_consensus.cc:2454] T f7264487a83e4574a7946e53f5dc8612 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:10:07.569960 30343 leader_election.cc:302] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:10:07.570237 11240 raft_consensus.cc:2781] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:07.570302 11240 raft_consensus.cc:482] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.570339 11240 raft_consensus.cc:3037] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:07.571017 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e03e76474584273ae050595a2286559" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:10:07.571133 32684 raft_consensus.cc:3037] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:07.572145 32684 raft_consensus.cc:2454] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 2.
I20240501 14:10:07.572479 30210 leader_election.cc:302] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, db935501e9104de5b15c8780681d6c38; no voters: 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.572674 11711 raft_consensus.cc:2781] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:07.572749 11711 raft_consensus.cc:686] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:10:07.572880 11711 consensus_queue.cc:237] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:07.573467 11694 raft_consensus.cc:482] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.573563 11694 raft_consensus.cc:504] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:07.573993 11694 leader_election.cc:288] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:10:07.575311 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4669cad44f4a3c9e01c4bba3c14734" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:10:07.575415 32684 raft_consensus.cc:2454] T 8f4669cad44f4a3c9e01c4bba3c14734 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:10:07.575681 30079 leader_election.cc:302] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d35c16dcb506426993811bb2519d7db3, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:10:07.575973 10417 raft_consensus.cc:2781] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:07.576038 10417 raft_consensus.cc:482] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.576079 10417 raft_consensus.cc:3037] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:07.577324 10417 raft_consensus.cc:504] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:07.577674 10417 leader_election.cc:288] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
W20240501 14:10:07.580294 30082 leader_election.cc:334] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.581666 29864 catalog_manager.cc:5484] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 1 to 2, leader changed from 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 2 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:10:07.583451 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4669cad44f4a3c9e01c4bba3c14734" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:10:07.583554 32684 raft_consensus.cc:3037] T 8f4669cad44f4a3c9e01c4bba3c14734 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:10:07.584337 30082 leader_election.cc:334] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.572170 11240 raft_consensus.cc:504] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:07.585004 11240 leader_election.cc:288] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:07.585534 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7264487a83e4574a7946e53f5dc8612" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:10:07.585646 30277 raft_consensus.cc:3037] T f7264487a83e4574a7946e53f5dc8612 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:07.586735 30277 raft_consensus.cc:2454] T f7264487a83e4574a7946e53f5dc8612 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 2.
I20240501 14:10:07.588510 30343 leader_election.cc:302] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:10:07.589098 11240 raft_consensus.cc:2781] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:07.589289 11240 raft_consensus.cc:686] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:10:07.589437 11240 consensus_queue.cc:237] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
W20240501 14:10:07.590090 30344 leader_election.cc:334] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:07.592715 30344 leader_election.cc:334] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.593106 11641 raft_consensus.cc:482] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.593223 11641 raft_consensus.cc:504] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:07.593624 11641 leader_election.cc:288] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:07.593757 32684 raft_consensus.cc:2454] T 8f4669cad44f4a3c9e01c4bba3c14734 P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 2.
I20240501 14:10:07.594267 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21efaac3a3d643679c5abaaddb9e5a3a" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:10:07.594389 30277 raft_consensus.cc:2454] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 1.
I20240501 14:10:07.594473 30079 leader_election.cc:302] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d35c16dcb506426993811bb2519d7db3, db935501e9104de5b15c8780681d6c38; no voters: 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.594651 11694 raft_consensus.cc:2781] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:07.594761 11694 raft_consensus.cc:686] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [term 2 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:10:07.594854 32620 leader_election.cc:302] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:10:07.594887 11694 consensus_queue.cc:237] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:07.595029 11641 raft_consensus.cc:2781] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:07.595103 11641 raft_consensus.cc:482] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.595182 11641 raft_consensus.cc:3037] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:07.596449 11641 raft_consensus.cc:504] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:07.596781 11641 leader_election.cc:288] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 2 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:07.597095 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21efaac3a3d643679c5abaaddb9e5a3a" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:10:07.597193 30277 raft_consensus.cc:3037] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:10:07.596796 32621 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.597818 29866 catalog_manager.cc:5484] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 1 to 2, leader changed from 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 2 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:07.598165 30277 raft_consensus.cc:2454] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db935501e9104de5b15c8780681d6c38 in term 2.
I20240501 14:10:07.598778 32620 leader_election.cc:302] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:10:07.598932 11641 raft_consensus.cc:2781] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:07.599010 11641 raft_consensus.cc:686] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [term 2 LEADER]: Becoming Leader. State: Replica: db935501e9104de5b15c8780681d6c38, State: Running, Role: LEADER
W20240501 14:10:07.599072 32621 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.599153 11641 consensus_queue.cc:237] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:07.602993 29870 catalog_manager.cc:5484] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 reported cstate change: term changed from 1 to 2, leader changed from 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) to db935501e9104de5b15c8780681d6c38 (127.22.252.135). New cstate: current_term: 2 leader_uuid: "db935501e9104de5b15c8780681d6c38" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:10:07.603663 11694 raft_consensus.cc:482] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.603809 11694 raft_consensus.cc:504] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:07.604477 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb8f294e9704c549e1e0590b1429fe5" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:10:07.604597 349 raft_consensus.cc:2454] T 7cb8f294e9704c549e1e0590b1429fe5 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
I20240501 14:10:07.604933 30081 leader_election.cc:302] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, d35c16dcb506426993811bb2519d7db3; no voters:
I20240501 14:10:07.605075 10417 raft_consensus.cc:2781] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:07.605144 10417 raft_consensus.cc:482] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.605190 10417 raft_consensus.cc:3037] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:07.605368 11694 leader_election.cc:288] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:10:07.601428 29865 catalog_manager.cc:5484] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 1 to 2, leader changed from 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 2 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:07.606339 10417 raft_consensus.cc:504] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
W20240501 14:10:07.606585 30082 leader_election.cc:334] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.606664 10417 leader_election.cc:288] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 election: Requested vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:10:07.606925 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb8f294e9704c549e1e0590b1429fe5" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:10:07.607036 349 raft_consensus.cc:3037] T 7cb8f294e9704c549e1e0590b1429fe5 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:07.608186 349 raft_consensus.cc:2454] T 7cb8f294e9704c549e1e0590b1429fe5 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 2.
W20240501 14:10:07.608608 30082 leader_election.cc:334] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.608709 30081 leader_election.cc:302] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, d35c16dcb506426993811bb2519d7db3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.608947 10417 raft_consensus.cc:2781] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:07.609035 10417 raft_consensus.cc:686] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [term 2 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:10:07.609169 10417 consensus_queue.cc:237] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:07.614225 29870 catalog_manager.cc:5484] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 1 to 2, leader changed from 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 2 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:07.616926 11204 raft_consensus.cc:482] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.617075 11204 raft_consensus.cc:504] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:07.617499 11204 leader_election.cc:288] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:10:07.619057 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce3093e24347f3a4e121af452c6361" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
W20240501 14:10:07.619110 32359 leader_election.cc:341] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Not found: Tablet not found: 89ce3093e24347f3a4e121af452c6361
W20240501 14:10:07.619567 32359 leader_election.cc:334] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.619632 32359 leader_election.cc:302] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3
I20240501 14:10:07.619776 11204 raft_consensus.cc:2726] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:07.622205 10880 raft_consensus.cc:482] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.622299 10880 raft_consensus.cc:504] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:07.622649 10880 leader_election.cc:288] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:10:07.622908 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce3093e24347f3a4e121af452c6361" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:10:07.623028 32415 raft_consensus.cc:2454] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
W20240501 14:10:07.623831 30344 leader_election.cc:334] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:07.624368 30344 leader_election.cc:341] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Not found: Tablet not found: 89ce3093e24347f3a4e121af452c6361
I20240501 14:10:07.624449 30344 leader_election.cc:302] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3
I20240501 14:10:07.624642 10880 raft_consensus.cc:2726] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:07.625234 11714 raft_consensus.cc:482] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.625370 11714 raft_consensus.cc:504] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:07.625761 11714 leader_election.cc:288] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:10:07.626271 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8ba266a8d0406bb93d6c578fe6c7cb" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:10:07.626420 32684 raft_consensus.cc:2454] T aa8ba266a8d0406bb93d6c578fe6c7cb P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:10:07.626740 30210 leader_election.cc:302] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, db935501e9104de5b15c8780681d6c38; no voters:
W20240501 14:10:07.627794 30213 leader_election.cc:334] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.627947 11714 raft_consensus.cc:2781] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:07.628008 11714 raft_consensus.cc:482] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.628043 11714 raft_consensus.cc:3037] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:07.629138 11714 raft_consensus.cc:504] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:07.629472 11714 leader_election.cc:288] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:10:07.629853 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8ba266a8d0406bb93d6c578fe6c7cb" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:10:07.629971 32684 raft_consensus.cc:3037] T aa8ba266a8d0406bb93d6c578fe6c7cb P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:10:07.630280 30213 leader_election.cc:334] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.630934 32684 raft_consensus.cc:2454] T aa8ba266a8d0406bb93d6c578fe6c7cb P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 2.
I20240501 14:10:07.631337 30210 leader_election.cc:302] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, db935501e9104de5b15c8780681d6c38; no voters: 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.632043 11714 raft_consensus.cc:2781] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:07.632134 11714 raft_consensus.cc:686] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [term 2 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:10:07.633167 11714 consensus_queue.cc:237] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:07.636933 29870 catalog_manager.cc:5484] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 1 to 2, leader changed from 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 2 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:10:07.645591 10798 raft_consensus.cc:482] T 7534d5788f96488fbad5f6334f476224 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.645697 10798 raft_consensus.cc:504] T 7534d5788f96488fbad5f6334f476224 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:07.646086 10798 leader_election.cc:288] T 7534d5788f96488fbad5f6334f476224 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:07.648986 10798 raft_consensus.cc:482] T 0cc8946505474737a610de35ddf0f2f6 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.649081 10798 raft_consensus.cc:504] T 0cc8946505474737a610de35ddf0f2f6 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:07.649416 10798 leader_election.cc:288] T 0cc8946505474737a610de35ddf0f2f6 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:10:07.650374 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7534d5788f96488fbad5f6334f476224" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
W20240501 14:10:07.651587 32621 leader_election.cc:334] T 7534d5788f96488fbad5f6334f476224 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.651657 32621 leader_election.cc:302] T 7534d5788f96488fbad5f6334f476224 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db935501e9104de5b15c8780681d6c38; no voters: 1d0bb69611af4876a2b3b33c4766e13a, 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.653239 10798 raft_consensus.cc:2726] T 7534d5788f96488fbad5f6334f476224 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:07.654089 11711 raft_consensus.cc:482] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.654170 11711 raft_consensus.cc:504] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:07.654496 11711 leader_election.cc:288] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
W20240501 14:10:07.656777 30213 leader_election.cc:334] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.657502 30255 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:07.660948 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cc8946505474737a610de35ddf0f2f6" candidate_uuid: "db935501e9104de5b15c8780681d6c38" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
W20240501 14:10:07.661314 32621 leader_election.cc:334] T 0cc8946505474737a610de35ddf0f2f6 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.662629 32618 leader_election.cc:302] T 0cc8946505474737a610de35ddf0f2f6 P db935501e9104de5b15c8780681d6c38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db935501e9104de5b15c8780681d6c38; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, d1c217469ae4478ebd49e4ed9f7352d7
I20240501 14:10:07.664297 10798 raft_consensus.cc:2726] T 0cc8946505474737a610de35ddf0f2f6 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:07.664186 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7534d5788f96488fbad5f6334f476224" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:10:07.664516 32684 raft_consensus.cc:2454] T 7534d5788f96488fbad5f6334f476224 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:10:07.664726 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:07.664788 30210 leader_election.cc:302] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, db935501e9104de5b15c8780681d6c38; no voters: 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.665258 11714 raft_consensus.cc:2781] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:07.665323 11714 raft_consensus.cc:482] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.665366 11714 raft_consensus.cc:3037] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:07.668201 11714 raft_consensus.cc:504] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:07.669541 11714 leader_election.cc:288] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:10:07.670603 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:07.670825 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7534d5788f96488fbad5f6334f476224" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:10:07.670934 32683 raft_consensus.cc:3037] T 7534d5788f96488fbad5f6334f476224 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:07.670917 11204 raft_consensus.cc:482] T 9e5188c15666488c99717f68acfeeea4 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.670979 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:07.671012 11204 raft_consensus.cc:504] T 9e5188c15666488c99717f68acfeeea4 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:07.671422 11204 leader_election.cc:288] T 9e5188c15666488c99717f68acfeeea4 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:07.672010 32683 raft_consensus.cc:2454] T 7534d5788f96488fbad5f6334f476224 P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 2.
I20240501 14:10:07.672314 30210 leader_election.cc:302] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:10:07.672664 11711 raft_consensus.cc:2781] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:07.672757 11711 raft_consensus.cc:686] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:10:07.672896 11711 consensus_queue.cc:237] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:07.677289 32401 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:07.678453 30386 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:10:07.681443 327 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:10:07.681737 11204 raft_consensus.cc:482] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.681831 11204 raft_consensus.cc:504] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:07.682224 11204 leader_election.cc:288] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:10:07.682837 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22bb82c5d65640128fc3271212aafb60" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:10:07.682963 30407 raft_consensus.cc:2454] T 22bb82c5d65640128fc3271212aafb60 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
I20240501 14:10:07.683249 32356 leader_election.cc:302] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
W20240501 14:10:07.683327 30213 leader_election.cc:334] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:07.683526 32359 leader_election.cc:334] T 9e5188c15666488c99717f68acfeeea4 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.683640 11204 raft_consensus.cc:2781] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:07.683688 11204 raft_consensus.cc:482] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.683723 11204 raft_consensus.cc:3037] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:07.683759 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5188c15666488c99717f68acfeeea4" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:10:07.684135 32358 leader_election.cc:302] T 9e5188c15666488c99717f68acfeeea4 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3; no voters: 1d0bb69611af4876a2b3b33c4766e13a, 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.684360 10978 raft_consensus.cc:2726] T 9e5188c15666488c99717f68acfeeea4 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:07.684967 11204 raft_consensus.cc:504] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:07.685286 11204 leader_election.cc:288] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:10:07.685624 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22bb82c5d65640128fc3271212aafb60" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:10:07.685724 30406 raft_consensus.cc:3037] T 22bb82c5d65640128fc3271212aafb60 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:10:07.686684 32359 leader_election.cc:334] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.686719 30406 raft_consensus.cc:2454] T 22bb82c5d65640128fc3271212aafb60 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 2.
I20240501 14:10:07.686988 32356 leader_election.cc:302] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
W20240501 14:10:07.688426 32359 leader_election.cc:334] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.688552 10978 raft_consensus.cc:2781] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:07.688704 10978 raft_consensus.cc:686] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [term 2 LEADER]: Becoming Leader. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Running, Role: LEADER
I20240501 14:10:07.688876 10978 consensus_queue.cc:237] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:07.689417 11064 raft_consensus.cc:482] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.689502 11064 raft_consensus.cc:504] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:10:07.689836 11064 leader_election.cc:288] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:10:07.690515 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46319e2fdff439e969092ead5d7e11e" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:10:07.690634 30145 raft_consensus.cc:2454] T a46319e2fdff439e969092ead5d7e11e P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
W20240501 14:10:07.691102 32753 leader_election.cc:334] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.691243 32753 leader_election.cc:302] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, d35c16dcb506426993811bb2519d7db3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.692193 29870 catalog_manager.cc:5484] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: term changed from 1 to 2, leader changed from 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) to a91029b90bf9415a89a2f70736759ad3 (127.22.252.133). New cstate: current_term: 2 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:07.692750 10620 raft_consensus.cc:2781] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:07.692806 10620 raft_consensus.cc:482] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.692843 10620 raft_consensus.cc:3037] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:07.693902 10620 raft_consensus.cc:504] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:10:07.694222 10620 leader_election.cc:288] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 election: Requested vote from peers d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
I20240501 14:10:07.697456 29864 catalog_manager.cc:5484] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 1 to 2, leader changed from 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 2 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:10:07.706313 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46319e2fdff439e969092ead5d7e11e" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:10:07.706429 30145 raft_consensus.cc:3037] T a46319e2fdff439e969092ead5d7e11e P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:07.707412 30145 raft_consensus.cc:2454] T a46319e2fdff439e969092ead5d7e11e P d35c16dcb506426993811bb2519d7db3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 2.
W20240501 14:10:07.708853 32753 leader_election.cc:334] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.709022 32753 leader_election.cc:302] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, d35c16dcb506426993811bb2519d7db3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.709206 11064 raft_consensus.cc:2781] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:07.709304 11064 raft_consensus.cc:686] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:10:07.709429 11064 consensus_queue.cc:237] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:10:07.710633 11714 raft_consensus.cc:482] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.710723 11714 raft_consensus.cc:504] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:07.711041 11714 leader_election.cc:288] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:10:07.712318 29870 catalog_manager.cc:5484] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 1 to 2, leader changed from 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 2 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } }
I20240501 14:10:07.712798 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5188c15666488c99717f68acfeeea4" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:10:07.712932 32415 raft_consensus.cc:2454] T 9e5188c15666488c99717f68acfeeea4 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:10:07.713687 30212 leader_election.cc:302] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:10:07.713825 11714 raft_consensus.cc:2781] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:07.713861 11714 raft_consensus.cc:482] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.713894 11714 raft_consensus.cc:3037] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:07.714933 11714 raft_consensus.cc:504] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
W20240501 14:10:07.720860 30213 leader_election.cc:334] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.721400 11714 leader_election.cc:288] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:10:07.722047 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5188c15666488c99717f68acfeeea4" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:10:07.722177 32415 raft_consensus.cc:3037] T 9e5188c15666488c99717f68acfeeea4 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:10:07.722545 30213 leader_election.cc:334] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.723237 32415 raft_consensus.cc:2454] T 9e5188c15666488c99717f68acfeeea4 P a91029b90bf9415a89a2f70736759ad3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 2.
I20240501 14:10:07.723582 30212 leader_election.cc:302] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.724610 11166 raft_consensus.cc:2781] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:07.724707 11166 raft_consensus.cc:686] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:10:07.724823 11166 consensus_queue.cc:237] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:07.726377 11714 raft_consensus.cc:482] T a63e9bcd27234f2f9a9e0b268fd73486 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.726469 11714 raft_consensus.cc:504] T a63e9bcd27234f2f9a9e0b268fd73486 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:10:07.726754 11714 leader_election.cc:288] T a63e9bcd27234f2f9a9e0b268fd73486 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:10:07.727095 349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63e9bcd27234f2f9a9e0b268fd73486" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
W20240501 14:10:07.727916 30213 leader_election.cc:334] T a63e9bcd27234f2f9a9e0b268fd73486 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.727982 30213 leader_election.cc:302] T a63e9bcd27234f2f9a9e0b268fd73486 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, 9ecd907cf5634ac98f3f54f3dbec6849
I20240501 14:10:07.728099 11714 raft_consensus.cc:2726] T a63e9bcd27234f2f9a9e0b268fd73486 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:07.731731 29870 catalog_manager.cc:5484] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 1 to 2, leader changed from 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 2 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:10:07.732738 10880 raft_consensus.cc:482] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.732841 10880 raft_consensus.cc:504] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:07.733186 10880 leader_election.cc:288] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:10:07.733672 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cc8946505474737a610de35ddf0f2f6" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38" is_pre_election: true
I20240501 14:10:07.733795 32684 raft_consensus.cc:2454] T 0cc8946505474737a610de35ddf0f2f6 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:10:07.734038 30341 leader_election.cc:302] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, db935501e9104de5b15c8780681d6c38; no voters:
I20240501 14:10:07.734202 10880 raft_consensus.cc:2781] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:07.734246 10880 raft_consensus.cc:482] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.734284 10880 raft_consensus.cc:3037] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:10:07.734328 30344 leader_election.cc:334] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.735327 10880 raft_consensus.cc:504] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:07.735631 10880 leader_election.cc:288] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:10:07.736312 32684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cc8946505474737a610de35ddf0f2f6" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db935501e9104de5b15c8780681d6c38"
I20240501 14:10:07.736415 32684 raft_consensus.cc:3037] T 0cc8946505474737a610de35ddf0f2f6 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:10:07.736833 30344 leader_election.cc:334] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.740088 32684 raft_consensus.cc:2454] T 0cc8946505474737a610de35ddf0f2f6 P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 2.
I20240501 14:10:07.740393 30341 leader_election.cc:302] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7, db935501e9104de5b15c8780681d6c38; no voters: 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.740641 10880 raft_consensus.cc:2781] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:07.740989 10880 raft_consensus.cc:686] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:10:07.741379 10880 consensus_queue.cc:237] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:07.744529 29870 catalog_manager.cc:5484] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 1 to 2, leader changed from 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 2 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:07.763391 10620 raft_consensus.cc:482] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.763513 10620 raft_consensus.cc:504] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:07.763868 10620 leader_election.cc:288] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:10:07.764372 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd36b13ca2274fc19b9d32d202e94f83" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
I20240501 14:10:07.764506 30145 raft_consensus.cc:2454] T bd36b13ca2274fc19b9d32d202e94f83 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
W20240501 14:10:07.764746 32753 leader_election.cc:334] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.764918 32753 leader_election.cc:302] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, d35c16dcb506426993811bb2519d7db3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.765269 10620 raft_consensus.cc:2781] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:07.765331 10620 raft_consensus.cc:482] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.765390 10620 raft_consensus.cc:3037] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:07.766256 10620 raft_consensus.cc:504] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:07.766595 10620 leader_election.cc:288] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:10:07.767324 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd36b13ca2274fc19b9d32d202e94f83" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3"
I20240501 14:10:07.767441 30145 raft_consensus.cc:3037] T bd36b13ca2274fc19b9d32d202e94f83 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:10:07.767450 32753 leader_election.cc:334] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.768363 30145 raft_consensus.cc:2454] T bd36b13ca2274fc19b9d32d202e94f83 P d35c16dcb506426993811bb2519d7db3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 2.
I20240501 14:10:07.768585 32753 leader_election.cc:302] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849, d35c16dcb506426993811bb2519d7db3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.768750 10620 raft_consensus.cc:2781] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:07.768822 10620 raft_consensus.cc:686] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:10:07.768939 10620 consensus_queue.cc:237] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:07.771253 10620 raft_consensus.cc:482] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.771342 10620 raft_consensus.cc:504] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:10:07.771555 29870 catalog_manager.cc:5484] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 1 to 2, leader changed from 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 2 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:07.771715 10620 leader_election.cc:288] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:07.772015 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63e9bcd27234f2f9a9e0b268fd73486" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:10:07.772140 30276 raft_consensus.cc:2454] T a63e9bcd27234f2f9a9e0b268fd73486 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:10:07.772550 32752 leader_election.cc:302] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
I20240501 14:10:07.772850 11064 raft_consensus.cc:2781] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:07.772897 11064 raft_consensus.cc:482] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.772935 11064 raft_consensus.cc:3037] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:10:07.773931 32753 leader_election.cc:334] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.775205 11064 raft_consensus.cc:504] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:10:07.775553 11064 leader_election.cc:288] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
W20240501 14:10:07.776388 32753 leader_election.cc:334] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.776489 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63e9bcd27234f2f9a9e0b268fd73486" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:10:07.776595 30276 raft_consensus.cc:3037] T a63e9bcd27234f2f9a9e0b268fd73486 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:07.777613 30276 raft_consensus.cc:2454] T a63e9bcd27234f2f9a9e0b268fd73486 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 2.
I20240501 14:10:07.778339 32752 leader_election.cc:302] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 9ecd907cf5634ac98f3f54f3dbec6849; no voters: 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.778481 11064 raft_consensus.cc:2781] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:07.778566 11064 raft_consensus.cc:686] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:10:07.778695 11064 consensus_queue.cc:237] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:10:07.780807 29870 catalog_manager.cc:5484] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 1 to 2, leader changed from 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 2 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } }
I20240501 14:10:07.782919 11714 raft_consensus.cc:482] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.783017 11714 raft_consensus.cc:504] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:07.783368 11714 leader_election.cc:288] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:10:07.783797 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b816de53d6a74e798a3b4d8988165c54" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:10:07.783914 32415 raft_consensus.cc:2454] T b816de53d6a74e798a3b4d8988165c54 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:10:07.784134 32416 consensus_queue.cc:237] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:07.784179 30212 leader_election.cc:302] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:10:07.784375 11714 raft_consensus.cc:2781] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:07.784432 11714 raft_consensus.cc:482] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:07.784482 11714 raft_consensus.cc:3037] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:07.785405 30407 raft_consensus.cc:1261] T ce04f87c84f54f1aaabb7cf10afe5b50 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:07.785461 11714 raft_consensus.cc:504] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:07.785778 10978 consensus_queue.cc:1035] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:07.785809 11714 leader_election.cc:288] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: Requested vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
W20240501 14:10:07.785969 30213 leader_election.cc:334] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.786554 30276 raft_consensus.cc:1261] T ce04f87c84f54f1aaabb7cf10afe5b50 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:07.786940 10978 consensus_queue.cc:1035] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 14:10:07.787267 30213 leader_election.cc:334] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:07.788839 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b816de53d6a74e798a3b4d8988165c54" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:10:07.788942 32416 raft_consensus.cc:3037] T b816de53d6a74e798a3b4d8988165c54 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:07.788924 11833 raft_consensus.cc:2932] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:07.789335 30276 raft_consensus.cc:2932] T ce04f87c84f54f1aaabb7cf10afe5b50 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:07.789964 32416 raft_consensus.cc:2454] T b816de53d6a74e798a3b4d8988165c54 P a91029b90bf9415a89a2f70736759ad3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 2.
I20240501 14:10:07.790251 30212 leader_election.cc:302] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:10:07.790453 11711 raft_consensus.cc:2781] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:07.790560 11711 raft_consensus.cc:686] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:10:07.790669 11711 consensus_queue.cc:237] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:07.792132 30407 raft_consensus.cc:2932] T ce04f87c84f54f1aaabb7cf10afe5b50 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:07.792402 29870 catalog_manager.cc:5484] T ce04f87c84f54f1aaabb7cf10afe5b50 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index 2 to 3, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:07.795346 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet ce04f87c84f54f1aaabb7cf10afe5b50 with delete_type TABLET_DATA_TOMBSTONED (TS 93f2518fbadf4f9991cda93afacb363e not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46100
I20240501 14:10:07.795651 11859 tablet_replica.cc:330] T ce04f87c84f54f1aaabb7cf10afe5b50 P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:10:07.795820 11859 raft_consensus.cc:2229] T ce04f87c84f54f1aaabb7cf10afe5b50 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:07.795980 11859 raft_consensus.cc:2258] T ce04f87c84f54f1aaabb7cf10afe5b50 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:07.796440 29864 catalog_manager.cc:5484] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 1 to 2, leader changed from 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134) to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 2 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:07.796536 11859 ts_tablet_manager.cc:1875] T ce04f87c84f54f1aaabb7cf10afe5b50 P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:07.799623 11859 ts_tablet_manager.cc:1888] T ce04f87c84f54f1aaabb7cf10afe5b50 P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:10:07.799714 11859 log.cc:1196] T ce04f87c84f54f1aaabb7cf10afe5b50 P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/ce04f87c84f54f1aaabb7cf10afe5b50
I20240501 14:10:07.800482 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet ce04f87c84f54f1aaabb7cf10afe5b50 (table rebalancer_extra_table_28 [id=1e152e3403f94ab998a842bc24585ae1]) successfully deleted
I20240501 14:10:07.873234 30407 raft_consensus.cc:1261] T 9cf197259d9648b5a742dcbb74e7f89b P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:10:07.873654 11711 consensus_queue.cc:1035] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:10:07.886409 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:07.888878 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:07.914532 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:07.929924 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:07.930236 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:07.934258 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:07.935318 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:07.935930 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:07.943478 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:07.945940 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:08.027396 30276 raft_consensus.cc:1261] T f7264487a83e4574a7946e53f5dc8612 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:10:08.027894 11240 consensus_queue.cc:1035] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:10:08.031256 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:08.033768 32621 consensus_peers.cc:485] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:08.053674 30276 raft_consensus.cc:1261] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Refusing update from remote peer db935501e9104de5b15c8780681d6c38: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:10:08.054114 10798 consensus_queue.cc:1035] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:10:08.085772 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:08.088709 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:08.089043 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:08.093866 32684 raft_consensus.cc:1261] T 1e03e76474584273ae050595a2286559 P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:10:08.094237 11166 consensus_queue.cc:1035] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:10:08.094586 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:08.101318 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:08.106020 32416 raft_consensus.cc:1261] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3 [term 2 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:10:08.106578 11145 consensus_queue.cc:1035] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:10:08.113845 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:08.144637 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:08.149233 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:08.156605 347 raft_consensus.cc:1261] T 7cb8f294e9704c549e1e0590b1429fe5 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:10:08.157083 11694 consensus_queue.cc:1035] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:08.158849 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:08.161846 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:08.163172 32684 raft_consensus.cc:1261] T 7534d5788f96488fbad5f6334f476224 P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:10:08.164081 11714 consensus_queue.cc:1035] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:10:08.170459 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:08.170799 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:08.175666 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:08.183364 30406 raft_consensus.cc:1261] T 22bb82c5d65640128fc3271212aafb60 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:10:08.183807 11833 consensus_queue.cc:1035] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:08.184943 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:08.192703 32683 raft_consensus.cc:1261] T 8f4669cad44f4a3c9e01c4bba3c14734 P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:10:08.193132 10417 consensus_queue.cc:1035] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:08.211532 32683 raft_consensus.cc:1261] T 0cc8946505474737a610de35ddf0f2f6 P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:10:08.211882 11240 consensus_queue.cc:1035] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:10:08.216630 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:08.217094 32416 raft_consensus.cc:1261] T b816de53d6a74e798a3b4d8988165c54 P a91029b90bf9415a89a2f70736759ad3 [term 2 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:10:08.217595 11714 consensus_queue.cc:1035] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:08.223709 32683 raft_consensus.cc:1261] T aa8ba266a8d0406bb93d6c578fe6c7cb P db935501e9104de5b15c8780681d6c38 [term 2 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240501 14:10:08.225315 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:08.226626 11714 consensus_queue.cc:1035] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:10:08.232318 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:08.251415 30145 raft_consensus.cc:1261] T a46319e2fdff439e969092ead5d7e11e P d35c16dcb506426993811bb2519d7db3 [term 2 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:10:08.251884 10620 consensus_queue.cc:1035] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:08.256779 30276 raft_consensus.cc:1261] T a63e9bcd27234f2f9a9e0b268fd73486 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:10:08.257464 10620 consensus_queue.cc:1035] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:10:08.263218 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:08.267024 32416 raft_consensus.cc:1261] T 9e5188c15666488c99717f68acfeeea4 P a91029b90bf9415a89a2f70736759ad3 [term 2 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:10:08.267397 11714 consensus_queue.cc:1035] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:10:08.271168 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:08.284531 30014 consensus_queue.cc:237] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:08.285868 32684 raft_consensus.cc:1261] T 89d3e17f11a4460fa0d0a9a0b8342a66 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:08.285993 32416 raft_consensus.cc:1261] T 89d3e17f11a4460fa0d0a9a0b8342a66 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:08.286294 11145 consensus_queue.cc:1035] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:08.286482 11063 consensus_queue.cc:1035] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:08.288628 11145 raft_consensus.cc:2932] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:10:08.289167 32416 raft_consensus.cc:2932] T 89d3e17f11a4460fa0d0a9a0b8342a66 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:10:08.289333 32684 raft_consensus.cc:2932] T 89d3e17f11a4460fa0d0a9a0b8342a66 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:10:08.293637 29870 catalog_manager.cc:5484] T 89d3e17f11a4460fa0d0a9a0b8342a66 P db935501e9104de5b15c8780681d6c38 reported cstate change: config changed from index 2 to 3, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:10:08.297276 30255 tablet_service.cc:1514] Processing DeleteTablet for tablet 89d3e17f11a4460fa0d0a9a0b8342a66 with delete_type TABLET_DATA_TOMBSTONED (TS 1d0bb69611af4876a2b3b33c4766e13a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44464
I20240501 14:10:08.298388 12012 tablet_replica.cc:330] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:10:08.298763 12012 raft_consensus.cc:2229] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:08.298964 12012 raft_consensus.cc:2258] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:08.299434 12012 ts_tablet_manager.cc:1875] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:08.302829 12012 ts_tablet_manager.cc:1888] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:10:08.302913 12012 log.cc:1196] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/89d3e17f11a4460fa0d0a9a0b8342a66
I20240501 14:10:08.303511 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet 89d3e17f11a4460fa0d0a9a0b8342a66 (table rebalancer_extra_table_40 [id=b95e6b33a5404ce981d2d09cf6e50f40]) successfully deleted
I20240501 14:10:08.315713 30145 raft_consensus.cc:1261] T bd36b13ca2274fc19b9d32d202e94f83 P d35c16dcb506426993811bb2519d7db3 [term 2 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:10:08.316116 10620 consensus_queue.cc:1035] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:08.421367 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:08.421941 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:08.424901 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:08.426250 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:08.432935 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:08.441874 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:08.444478 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:10:08.453567 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:08.462013 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:10:08.592303 32621 consensus_peers.cc:485] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:08.623291 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:08.626525 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:08.628953 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:08.639380 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:08.645824 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:08.646628 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:08.648347 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:10:08.742230 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:08.755375 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:08.784368 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:08.836081 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:08.853406 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:08.854687 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:08.856814 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:08.861608 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:08.864192 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:08.872762 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:08.877159 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:08.878528 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:08.879588 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:08.883271 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:08.899667 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:08.901975 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:08.911561 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:08.930840 32621 consensus_peers.cc:485] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:08.933866 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:08.948556 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:08.965968 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:08.986440 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:09.028321 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:09.032001 32621 consensus_peers.cc:485] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:09.049060 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:09.050707 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:09.055891 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:09.058761 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:09.058943 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:09.061300 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:09.061311 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:09.062131 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:09.066550 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:09.086642 32621 consensus_peers.cc:485] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:09.098776 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:10:09.119823 32621 consensus_peers.cc:485] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:09.121894 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:09.139868 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:09.152698 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:09.226647 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:09.235042 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:09.236438 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:09.236893 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:09.270888 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:09.273588 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:09.281713 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:09.283013 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:09.288921 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:09.295472 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:09.301654 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:09.304845 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:09.308718 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:10:09.324649 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:09.327397 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:09.328415 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:09.328814 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:10:09.334451 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:09.418540 32621 consensus_peers.cc:485] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:09.419945 10880 raft_consensus.cc:482] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:09.420053 10880 raft_consensus.cc:504] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:09.420565 10880 leader_election.cc:288] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
W20240501 14:10:09.421756 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:09.421787 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce3093e24347f3a4e121af452c6361" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:10:09.421923 32416 raft_consensus.cc:2454] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
W20240501 14:10:09.421927 30344 leader_election.cc:341] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Not found: Tablet not found: 89ce3093e24347f3a4e121af452c6361
W20240501 14:10:09.422271 30344 leader_election.cc:334] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:09.422326 30344 leader_election.cc:302] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3
I20240501 14:10:09.422461 10880 raft_consensus.cc:2726] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:09.430562 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:09.486618 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:09.489634 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:09.494786 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:09.495638 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:09.496490 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:09.500135 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:09.510623 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:09.530050 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:09.730234 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:09.732545 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:09.739235 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:09.739615 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:09.742524 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:09.747963 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:09.748531 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:09.821967 30276 consensus_queue.cc:237] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:09.823228 32415 raft_consensus.cc:1261] T 75b67ccb9a334b25ae71b553a2a84a02 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:09.823210 32684 raft_consensus.cc:1261] T 75b67ccb9a334b25ae71b553a2a84a02 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:09.823578 11714 consensus_queue.cc:1035] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:09.823911 11983 consensus_queue.cc:1035] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:09.825554 11714 raft_consensus.cc:2932] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:09.828004 32415 raft_consensus.cc:2932] T 75b67ccb9a334b25ae71b553a2a84a02 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:09.830052 29870 catalog_manager.cc:5484] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: config changed from index 6 to 7, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:09.832195 32683 raft_consensus.cc:2932] T 75b67ccb9a334b25ae71b553a2a84a02 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:09.834618 30386 tablet_service.cc:1514] Processing DeleteTablet for tablet 75b67ccb9a334b25ae71b553a2a84a02 with delete_type TABLET_DATA_TOMBSTONED (TS d1c217469ae4478ebd49e4ed9f7352d7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38118
I20240501 14:10:09.835359 12413 tablet_replica.cc:330] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7: stopping tablet replica
I20240501 14:10:09.835556 12413 raft_consensus.cc:2229] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:09.835819 12413 raft_consensus.cc:2258] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:09.836630 12413 ts_tablet_manager.cc:1875] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:09.840284 12413 ts_tablet_manager.cc:1888] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:10:09.840380 12413 log.cc:1196] T 75b67ccb9a334b25ae71b553a2a84a02 P d1c217469ae4478ebd49e4ed9f7352d7: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/wals/75b67ccb9a334b25ae71b553a2a84a02
I20240501 14:10:09.841230 29857 catalog_manager.cc:4830] TS d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): tablet 75b67ccb9a334b25ae71b553a2a84a02 (table rebalance_test_table_19 [id=00bf2d6ad5af4e6480096d5ceb4b1bb8]) successfully deleted
I20240501 14:10:09.870210 349 consensus_queue.cc:237] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:09.871768 32415 raft_consensus.cc:1261] T 7628284cb4484ee5b87173790179def0 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:09.871834 32683 raft_consensus.cc:1261] T 7628284cb4484ee5b87173790179def0 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:10:09.872205 32753 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Couldn't send request to peer d35c16dcb506426993811bb2519d7db3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7628284cb4484ee5b87173790179def0. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:09.872318 12441 consensus_queue.cc:1035] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:09.872488 12440 consensus_queue.cc:1035] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:09.874143 12441 raft_consensus.cc:2932] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:09.874586 32415 raft_consensus.cc:2932] T 7628284cb4484ee5b87173790179def0 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:09.876765 32683 raft_consensus.cc:2932] T 7628284cb4484ee5b87173790179def0 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:09.877717 29870 catalog_manager.cc:5484] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: config changed from index -1 to 2, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:09.896303 347 consensus_queue.cc:237] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:09.897878 32415 raft_consensus.cc:1261] T 440f03e9113e46f6ba4f019071ebdacb P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:09.898253 11064 consensus_queue.cc:1035] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:10:09.898465 32752 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): Couldn't send request to peer 93f2518fbadf4f9991cda93afacb363e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 440f03e9113e46f6ba4f019071ebdacb. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:09.898854 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:09.909379 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:10.260540 32621 consensus_peers.cc:485] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:10.275662 12462 ts_tablet_manager.cc:902] T 7628284cb4484ee5b87173790179def0 P d35c16dcb506426993811bb2519d7db3: Initiating tablet copy from peer 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:10:10.276144 12462 tablet_copy_client.cc:308] T 7628284cb4484ee5b87173790179def0 P d35c16dcb506426993811bb2519d7db3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.136:45937
I20240501 14:10:10.276603 369 tablet_copy_service.cc:140] P 9ecd907cf5634ac98f3f54f3dbec6849: Received BeginTabletCopySession request for tablet 7628284cb4484ee5b87173790179def0 from peer d35c16dcb506426993811bb2519d7db3 ({username='slave'} at 127.22.252.130:43657)
I20240501 14:10:10.276703 369 tablet_copy_service.cc:161] P 9ecd907cf5634ac98f3f54f3dbec6849: Beginning new tablet copy session on tablet 7628284cb4484ee5b87173790179def0 from peer d35c16dcb506426993811bb2519d7db3 at {username='slave'} at 127.22.252.130:43657: session id = d35c16dcb506426993811bb2519d7db3-7628284cb4484ee5b87173790179def0
I20240501 14:10:10.277704 369 tablet_copy_source_session.cc:205] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:10.278648 12462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7628284cb4484ee5b87173790179def0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:10.280598 12462 tablet_copy_client.cc:783] T 7628284cb4484ee5b87173790179def0 P d35c16dcb506426993811bb2519d7db3: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:10.280745 12462 tablet_copy_client.cc:647] T 7628284cb4484ee5b87173790179def0 P d35c16dcb506426993811bb2519d7db3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:10.281796 12462 tablet_copy_client.cc:523] T 7628284cb4484ee5b87173790179def0 P d35c16dcb506426993811bb2519d7db3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:10.282892 12462 tablet_bootstrap.cc:492] T 7628284cb4484ee5b87173790179def0 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:10:10.285706 12462 tablet_bootstrap.cc:492] T 7628284cb4484ee5b87173790179def0 P d35c16dcb506426993811bb2519d7db3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:10.285890 12462 tablet_bootstrap.cc:492] T 7628284cb4484ee5b87173790179def0 P d35c16dcb506426993811bb2519d7db3: Bootstrap complete.
I20240501 14:10:10.286001 12462 ts_tablet_manager.cc:1367] T 7628284cb4484ee5b87173790179def0 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:10.286355 12462 raft_consensus.cc:348] T 7628284cb4484ee5b87173790179def0 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:10.286444 12462 raft_consensus.cc:729] T 7628284cb4484ee5b87173790179def0 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:10:10.286558 12462 consensus_queue.cc:260] T 7628284cb4484ee5b87173790179def0 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:10.286841 12462 ts_tablet_manager.cc:1398] T 7628284cb4484ee5b87173790179def0 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:10.287672 369 tablet_copy_service.cc:342] P 9ecd907cf5634ac98f3f54f3dbec6849: Request end of tablet copy session d35c16dcb506426993811bb2519d7db3-7628284cb4484ee5b87173790179def0 received from {username='slave'} at 127.22.252.130:43657
I20240501 14:10:10.287746 369 tablet_copy_service.cc:433] P 9ecd907cf5634ac98f3f54f3dbec6849: ending tablet copy session d35c16dcb506426993811bb2519d7db3-7628284cb4484ee5b87173790179def0 on tablet 7628284cb4484ee5b87173790179def0 with peer d35c16dcb506426993811bb2519d7db3
W20240501 14:10:10.291222 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:10.311326 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:10.354226 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:10.415164 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:10.480453 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:10.517009 12468 ts_tablet_manager.cc:902] T 440f03e9113e46f6ba4f019071ebdacb P 93f2518fbadf4f9991cda93afacb363e: Initiating tablet copy from peer 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:10:10.517520 12468 tablet_copy_client.cc:308] T 440f03e9113e46f6ba4f019071ebdacb P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.136:45937
I20240501 14:10:10.517992 369 tablet_copy_service.cc:140] P 9ecd907cf5634ac98f3f54f3dbec6849: Received BeginTabletCopySession request for tablet 440f03e9113e46f6ba4f019071ebdacb from peer 93f2518fbadf4f9991cda93afacb363e ({username='slave'} at 127.22.252.129:40869)
I20240501 14:10:10.518083 369 tablet_copy_service.cc:161] P 9ecd907cf5634ac98f3f54f3dbec6849: Beginning new tablet copy session on tablet 440f03e9113e46f6ba4f019071ebdacb from peer 93f2518fbadf4f9991cda93afacb363e at {username='slave'} at 127.22.252.129:40869: session id = 93f2518fbadf4f9991cda93afacb363e-440f03e9113e46f6ba4f019071ebdacb
I20240501 14:10:10.518993 369 tablet_copy_source_session.cc:205] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:10.519889 12468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 440f03e9113e46f6ba4f019071ebdacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:10.522333 12468 tablet_copy_client.cc:783] T 440f03e9113e46f6ba4f019071ebdacb P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:10.522501 12468 tablet_copy_client.cc:647] T 440f03e9113e46f6ba4f019071ebdacb P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:10.523447 12468 tablet_copy_client.cc:523] T 440f03e9113e46f6ba4f019071ebdacb P 93f2518fbadf4f9991cda93afacb363e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:10.524426 12468 tablet_bootstrap.cc:492] T 440f03e9113e46f6ba4f019071ebdacb P 93f2518fbadf4f9991cda93afacb363e: Bootstrap starting.
I20240501 14:10:10.527400 12468 tablet_bootstrap.cc:492] T 440f03e9113e46f6ba4f019071ebdacb P 93f2518fbadf4f9991cda93afacb363e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:10:10.527666 12468 tablet_bootstrap.cc:492] T 440f03e9113e46f6ba4f019071ebdacb P 93f2518fbadf4f9991cda93afacb363e: Bootstrap complete.
I20240501 14:10:10.527796 12468 ts_tablet_manager.cc:1367] T 440f03e9113e46f6ba4f019071ebdacb P 93f2518fbadf4f9991cda93afacb363e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:10.528180 12468 raft_consensus.cc:348] T 440f03e9113e46f6ba4f019071ebdacb P 93f2518fbadf4f9991cda93afacb363e [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:10:10.528432 12468 raft_consensus.cc:729] T 440f03e9113e46f6ba4f019071ebdacb P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Initialized, Role: FOLLOWER
I20240501 14:10:10.528541 12468 consensus_queue.cc:260] T 440f03e9113e46f6ba4f019071ebdacb P 93f2518fbadf4f9991cda93afacb363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:10.528802 12468 ts_tablet_manager.cc:1398] T 440f03e9113e46f6ba4f019071ebdacb P 93f2518fbadf4f9991cda93afacb363e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:10:10.529285 369 tablet_copy_service.cc:342] P 9ecd907cf5634ac98f3f54f3dbec6849: Request end of tablet copy session 93f2518fbadf4f9991cda93afacb363e-440f03e9113e46f6ba4f019071ebdacb received from {username='slave'} at 127.22.252.129:40869
I20240501 14:10:10.529361 369 tablet_copy_service.cc:433] P 9ecd907cf5634ac98f3f54f3dbec6849: ending tablet copy session 93f2518fbadf4f9991cda93afacb363e-440f03e9113e46f6ba4f019071ebdacb on tablet 440f03e9113e46f6ba4f019071ebdacb with peer 93f2518fbadf4f9991cda93afacb363e
W20240501 14:10:10.552703 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:10.572702 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:10.583359 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:10.606535 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:10.666071 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:10.682641 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:10.704923 30146 raft_consensus.cc:1203] T 7628284cb4484ee5b87173790179def0 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 14:10:10.709846 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:10.783157 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:10.822048 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:10.855778 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:10.857322 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:10.982260 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:10.994602 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.077499 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.094234 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:11.107228 30014 raft_consensus.cc:1203] T 440f03e9113e46f6ba4f019071ebdacb P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:10:11.108439 11064 raft_consensus.cc:2932] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:11.109989 11064 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:32794 (request call id 0) took 1214 ms. Trace:
I20240501 14:10:11.110080 11064 rpcz_store.cc:276] 0501 14:10:09.895963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 14:10:09.895998 (+ 35us) service_pool.cc:225] Handling call
0501 14:10:09.897093 (+ 1095us) log.cc:842] Serialized 505 byte log entry
0501 14:10:11.109973 (+1212880us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":210,"raft.queue_time_us":171,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":334,"spinlock_wait_cycles":1032704,"thread_start_us":67,"threads_started":1,"wal-append.queue_time_us":164,"wal-append.run_cpu_time_us":314,"wal-append.run_wall_time_us":1000722}
I20240501 14:10:11.110662 30014 raft_consensus.cc:2932] T 440f03e9113e46f6ba4f019071ebdacb P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:11.111099 32415 raft_consensus.cc:2932] T 440f03e9113e46f6ba4f019071ebdacb P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:11.113726 29870 catalog_manager.cc:5484] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: config changed from index -1 to 2, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) added. New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:11.134531 30014 consensus_queue.cc:237] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:11.135857 30146 raft_consensus.cc:1261] T f726c6a4929541c9b8ba7e29b757bb89 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:10:11.135996 29950 consensus_peers.cc:485] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e -> Peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): Couldn't send request to peer a91029b90bf9415a89a2f70736759ad3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f726c6a4929541c9b8ba7e29b757bb89. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:11.135972 32683 raft_consensus.cc:1261] T f726c6a4929541c9b8ba7e29b757bb89 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:11.136348 12502 consensus_queue.cc:1035] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:11.136564 12474 consensus_queue.cc:1035] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:11.138392 12502 raft_consensus.cc:2932] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:11.140194 32683 raft_consensus.cc:2932] T f726c6a4929541c9b8ba7e29b757bb89 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:11.142851 30146 raft_consensus.cc:2932] T f726c6a4929541c9b8ba7e29b757bb89 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:11.144160 29872 catalog_manager.cc:5484] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: config changed from index -1 to 2, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
W20240501 14:10:11.155311 32621 consensus_peers.cc:485] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.164008 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:11.285473 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:11.289820 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:11.294907 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:11.295907 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:11.297978 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:11.302886 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.305871 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:11.315382 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:11.321120 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.333045 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.333724 32621 consensus_peers.cc:485] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:11.337476 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:10:11.357245 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.370594 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.374821 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.415063 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.419306 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.435557 32621 consensus_peers.cc:485] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.444140 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.448813 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.506520 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.513448 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:11.522079 12617 ts_tablet_manager.cc:902] T f726c6a4929541c9b8ba7e29b757bb89 P a91029b90bf9415a89a2f70736759ad3: Initiating tablet copy from peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:10:11.523558 12617 tablet_copy_client.cc:308] T f726c6a4929541c9b8ba7e29b757bb89 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.129:41387
W20240501 14:10:11.523790 32621 consensus_peers.cc:485] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:11.525272 30035 tablet_copy_service.cc:140] P 93f2518fbadf4f9991cda93afacb363e: Received BeginTabletCopySession request for tablet f726c6a4929541c9b8ba7e29b757bb89 from peer a91029b90bf9415a89a2f70736759ad3 ({username='slave'} at 127.22.252.133:60917)
I20240501 14:10:11.525380 30035 tablet_copy_service.cc:161] P 93f2518fbadf4f9991cda93afacb363e: Beginning new tablet copy session on tablet f726c6a4929541c9b8ba7e29b757bb89 from peer a91029b90bf9415a89a2f70736759ad3 at {username='slave'} at 127.22.252.133:60917: session id = a91029b90bf9415a89a2f70736759ad3-f726c6a4929541c9b8ba7e29b757bb89
I20240501 14:10:11.526397 30035 tablet_copy_source_session.cc:205] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:11.526930 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:11.527524 12617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f726c6a4929541c9b8ba7e29b757bb89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:11.531131 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:11.531724 12617 tablet_copy_client.cc:783] T f726c6a4929541c9b8ba7e29b757bb89 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:11.531870 12617 tablet_copy_client.cc:647] T f726c6a4929541c9b8ba7e29b757bb89 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:11.539501 12617 tablet_copy_client.cc:523] T f726c6a4929541c9b8ba7e29b757bb89 P a91029b90bf9415a89a2f70736759ad3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:11.539634 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:11.541493 12617 tablet_bootstrap.cc:492] T f726c6a4929541c9b8ba7e29b757bb89 P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
I20240501 14:10:11.546211 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:11.547824 12617 tablet_bootstrap.cc:492] T f726c6a4929541c9b8ba7e29b757bb89 P a91029b90bf9415a89a2f70736759ad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:11.548099 12617 tablet_bootstrap.cc:492] T f726c6a4929541c9b8ba7e29b757bb89 P a91029b90bf9415a89a2f70736759ad3: Bootstrap complete.
I20240501 14:10:11.548249 12617 ts_tablet_manager.cc:1367] T f726c6a4929541c9b8ba7e29b757bb89 P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 14:10:11.548694 12617 raft_consensus.cc:348] T f726c6a4929541c9b8ba7e29b757bb89 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:11.548815 12617 raft_consensus.cc:729] T f726c6a4929541c9b8ba7e29b757bb89 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:10:11.548936 12617 consensus_queue.cc:260] T f726c6a4929541c9b8ba7e29b757bb89 P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:11.549224 12617 ts_tablet_manager.cc:1398] T f726c6a4929541c9b8ba7e29b757bb89 P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:11.549849 30035 tablet_copy_service.cc:342] P 93f2518fbadf4f9991cda93afacb363e: Request end of tablet copy session a91029b90bf9415a89a2f70736759ad3-f726c6a4929541c9b8ba7e29b757bb89 received from {username='slave'} at 127.22.252.133:60917
I20240501 14:10:11.549927 30035 tablet_copy_service.cc:433] P 93f2518fbadf4f9991cda93afacb363e: ending tablet copy session a91029b90bf9415a89a2f70736759ad3-f726c6a4929541c9b8ba7e29b757bb89 on tablet f726c6a4929541c9b8ba7e29b757bb89 with peer a91029b90bf9415a89a2f70736759ad3
I20240501 14:10:11.550266 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:11.559152 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.566751 32621 consensus_peers.cc:485] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:11.568100 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:11.568500 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:10:11.580407 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.595563 30344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240501 14:10:11.596436 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.600844 32359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 14:10:11.601475 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.603272 30082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:10:11.603868 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.615520 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.631395 30213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240501 14:10:11.631969 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:11.650303 349 consensus_queue.cc:237] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:11.651607 32684 raft_consensus.cc:1261] T 7628284cb4484ee5b87173790179def0 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:11.651619 30146 raft_consensus.cc:1261] T 7628284cb4484ee5b87173790179def0 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:11.652004 12477 consensus_queue.cc:1035] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:11.652199 11064 consensus_queue.cc:1035] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:11.653880 12441 raft_consensus.cc:2932] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:11.654382 32684 raft_consensus.cc:2932] T 7628284cb4484ee5b87173790179def0 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:11.654385 30146 raft_consensus.cc:2932] T 7628284cb4484ee5b87173790179def0 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:11.658216 29867 catalog_manager.cc:5484] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: config changed from index 2 to 3, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) evicted. New cstate: current_term: 1 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:11.661357 32401 tablet_service.cc:1514] Processing DeleteTablet for tablet 7628284cb4484ee5b87173790179def0 with delete_type TABLET_DATA_TOMBSTONED (TS a91029b90bf9415a89a2f70736759ad3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53090
I20240501 14:10:11.661648 12648 tablet_replica.cc:330] T 7628284cb4484ee5b87173790179def0 P a91029b90bf9415a89a2f70736759ad3: stopping tablet replica
I20240501 14:10:11.661824 12648 raft_consensus.cc:2229] T 7628284cb4484ee5b87173790179def0 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:11.661973 12648 raft_consensus.cc:2258] T 7628284cb4484ee5b87173790179def0 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:11.662369 12648 ts_tablet_manager.cc:1875] T 7628284cb4484ee5b87173790179def0 P a91029b90bf9415a89a2f70736759ad3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 14:10:11.664620 32753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 83 similar messages]
I20240501 14:10:11.664930 12648 ts_tablet_manager.cc:1888] T 7628284cb4484ee5b87173790179def0 P a91029b90bf9415a89a2f70736759ad3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:10:11.665002 12648 log.cc:1196] T 7628284cb4484ee5b87173790179def0 P a91029b90bf9415a89a2f70736759ad3: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/wals/7628284cb4484ee5b87173790179def0
I20240501 14:10:11.665515 29859 catalog_manager.cc:4830] TS a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): tablet 7628284cb4484ee5b87173790179def0 (table rebalancer_extra_table_26 [id=fa42a548a7f24eab9694087203fda9ba]) successfully deleted
W20240501 14:10:11.674489 32621 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 14:10:11.687077 29951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:10:11.687659 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.719492 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.719766 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.726315 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.742929 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.756661 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.769450 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.786803 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:11.792428 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:10:11.793251 32621 consensus_peers.cc:485] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:11.794335 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:11.795019 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:11.795114 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:11.795382 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:11.806100 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:11.813469 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:11.815989 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:11.991324 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:11.991554 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:11.997025 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:12.010970 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:12.011515 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:12.013267 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:12.015144 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:12.028601 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:12.049219 32415 raft_consensus.cc:1203] T f726c6a4929541c9b8ba7e29b757bb89 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:10:12.093961 32415 consensus_queue.cc:237] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:12.095427 30013 raft_consensus.cc:1261] T 42636717a44e4e3782037c48d8913307 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:12.095430 30406 raft_consensus.cc:1261] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:12.096026 11985 consensus_queue.cc:1035] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:12.096606 11985 consensus_queue.cc:1035] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:12.098335 12273 raft_consensus.cc:2932] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:12.098768 30013 raft_consensus.cc:2932] T 42636717a44e4e3782037c48d8913307 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:12.101892 30406 raft_consensus.cc:2932] T 42636717a44e4e3782037c48d8913307 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:12.102615 29870 catalog_manager.cc:5484] T 42636717a44e4e3782037c48d8913307 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index 2 to 3, VOTER 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131) evicted. New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:12.106762 30255 tablet_service.cc:1514] Processing DeleteTablet for tablet 42636717a44e4e3782037c48d8913307 with delete_type TABLET_DATA_TOMBSTONED (TS 1d0bb69611af4876a2b3b33c4766e13a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44464
I20240501 14:10:12.107192 12777 tablet_replica.cc:330] T 42636717a44e4e3782037c48d8913307 P 1d0bb69611af4876a2b3b33c4766e13a: stopping tablet replica
I20240501 14:10:12.107451 12777 raft_consensus.cc:2229] T 42636717a44e4e3782037c48d8913307 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:12.107620 12777 raft_consensus.cc:2258] T 42636717a44e4e3782037c48d8913307 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:12.108211 12777 ts_tablet_manager.cc:1875] T 42636717a44e4e3782037c48d8913307 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:12.110532 12777 ts_tablet_manager.cc:1888] T 42636717a44e4e3782037c48d8913307 P 1d0bb69611af4876a2b3b33c4766e13a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:10:12.110597 12777 log.cc:1196] T 42636717a44e4e3782037c48d8913307 P 1d0bb69611af4876a2b3b33c4766e13a: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/wals/42636717a44e4e3782037c48d8913307
I20240501 14:10:12.111335 29859 catalog_manager.cc:4830] TS 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103): tablet 42636717a44e4e3782037c48d8913307 (table rebalancer_extra_table_52 [id=eaceddc0ae4d496fa3ffc653930395b9]) successfully deleted
W20240501 14:10:12.114780 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:12.140298 10880 raft_consensus.cc:482] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:12.140426 10880 raft_consensus.cc:504] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:12.140894 10880 leader_election.cc:288] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:10:12.141178 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce3093e24347f3a4e121af452c6361" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:10:12.141353 32415 raft_consensus.cc:2454] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
W20240501 14:10:12.141855 30344 leader_election.cc:334] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:12.142045 30344 leader_election.cc:341] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Not found: Tablet not found: 89ce3093e24347f3a4e121af452c6361
I20240501 14:10:12.142127 30344 leader_election.cc:302] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3
I20240501 14:10:12.142673 10880 raft_consensus.cc:2726] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:12.243144 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:12.263403 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:12.266086 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:12.268638 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:12.270787 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:12.282467 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:12.283504 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:12.284871 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:12.286578 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:12.465725 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:12.466396 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:12.473582 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:12.480890 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:12.481614 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:12.487344 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:12.518594 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:10:12.697314 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:12.705727 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:12.707026 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:12.710146 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:12.714184 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:12.723544 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:12.723814 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:12.740980 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 14:10:12.764786 32621 consensus_peers.cc:485] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:12.808444 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:12.827813 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:12.855414 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:12.916152 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:12.916443 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:12.921752 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:12.923983 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:12.937995 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:12.950042 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:12.961169 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:12.981325 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:12.992585 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:13.052680 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:13.082433 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:13.104300 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:13.113519 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:13.123380 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:13.169324 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:13.171097 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:10:13.180177 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:13.182166 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:13.182569 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:13.186864 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:13.204826 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:13.224637 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:10:13.259105 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:13.268760 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:13.393846 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:13.398733 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:13.400856 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:10:13.404920 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:13.407099 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:13.407843 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:13.416601 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:13.416853 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:13.417853 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:13.508360 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:13.509027 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:13.550141 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:13.556845 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:13.595901 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:13.603627 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:13.606474 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:10:13.614367 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:13.618643 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:13.619609 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:13.624801 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:13.628549 32621 consensus_peers.cc:485] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:13.629222 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:13.635921 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:10:13.649304 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:13.680914 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:13.715564 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:13.726064 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:13.789352 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:13.809721 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:13.809818 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:13.810570 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:13.817014 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:10:13.819567 32621 consensus_peers.cc:485] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:13.820909 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:13.829295 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:13.831943 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:13.834043 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:13.836372 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:13.847112 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:13.847980 32621 consensus_peers.cc:485] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:13.866182 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:13.872956 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:13.877859 13095 raft_consensus.cc:482] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:13.877962 13095 raft_consensus.cc:504] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:13.878409 13095 leader_election.cc:288] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:10:13.878777 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce3093e24347f3a4e121af452c6361" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:10:13.878904 32415 raft_consensus.cc:2454] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
W20240501 14:10:13.880424 30344 leader_election.cc:341] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Not found: Tablet not found: 89ce3093e24347f3a4e121af452c6361
W20240501 14:10:13.881397 30344 leader_election.cc:334] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:13.881454 30344 leader_election.cc:302] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3
I20240501 14:10:13.881628 13095 raft_consensus.cc:2726] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:13.904148 32621 consensus_peers.cc:485] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:13.917781 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:13.917937 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:13.947052 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:13.955545 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:13.986549 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:14.012192 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:14.018612 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:14.018656 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:14.029826 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:14.030772 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:14.034818 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:14.035663 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:14.035902 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:14.052412 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:10:14.054133 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:14.110204 32621 consensus_peers.cc:485] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:14.113306 32415 consensus_queue.cc:237] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:14.114820 30013 raft_consensus.cc:1261] T c90d65f39c3540f48c0d85b28130e76a P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:10:14.115115 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357): Couldn't send request to peer d1c217469ae4478ebd49e4ed9f7352d7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c90d65f39c3540f48c0d85b28130e76a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:14.116001 11985 consensus_queue.cc:1035] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:10:14.116921 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:14.118366 32683 raft_consensus.cc:1261] T c90d65f39c3540f48c0d85b28130e76a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:14.118796 11985 consensus_queue.cc:1035] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:14.120746 11985 raft_consensus.cc:2932] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:14.121124 32683 raft_consensus.cc:2932] T c90d65f39c3540f48c0d85b28130e76a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:14.124332 30013 raft_consensus.cc:2932] T c90d65f39c3540f48c0d85b28130e76a P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:14.126747 29864 catalog_manager.cc:5484] T c90d65f39c3540f48c0d85b28130e76a P db935501e9104de5b15c8780681d6c38 reported cstate change: config changed from index -1 to 2, VOTER d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132) added. New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:14.141999 32415 consensus_queue.cc:237] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:14.143282 349 raft_consensus.cc:1261] T b26b6d19699047afaee57d6f2c887664 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:10:14.143569 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Couldn't send request to peer d35c16dcb506426993811bb2519d7db3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b26b6d19699047afaee57d6f2c887664. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:14.143647 13281 consensus_queue.cc:1035] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:10:14.144006 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:14.194584 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:14.210312 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:14.229353 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:14.256201 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:14.279223 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:14.337124 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:14.415937 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:14.426096 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:14.457402 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:14.503952 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:14.532394 32621 consensus_peers.cc:485] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:14.616292 13305 ts_tablet_manager.cc:902] T b26b6d19699047afaee57d6f2c887664 P d35c16dcb506426993811bb2519d7db3: Initiating tablet copy from peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:10:14.616880 13305 tablet_copy_client.cc:308] T b26b6d19699047afaee57d6f2c887664 P d35c16dcb506426993811bb2519d7db3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.133:46265
I20240501 14:10:14.617352 32443 tablet_copy_service.cc:140] P a91029b90bf9415a89a2f70736759ad3: Received BeginTabletCopySession request for tablet b26b6d19699047afaee57d6f2c887664 from peer d35c16dcb506426993811bb2519d7db3 ({username='slave'} at 127.22.252.130:47505)
I20240501 14:10:14.617453 32443 tablet_copy_service.cc:161] P a91029b90bf9415a89a2f70736759ad3: Beginning new tablet copy session on tablet b26b6d19699047afaee57d6f2c887664 from peer d35c16dcb506426993811bb2519d7db3 at {username='slave'} at 127.22.252.130:47505: session id = d35c16dcb506426993811bb2519d7db3-b26b6d19699047afaee57d6f2c887664
I20240501 14:10:14.618438 32443 tablet_copy_source_session.cc:205] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:14.619679 13305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26b6d19699047afaee57d6f2c887664. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:14.622125 13305 tablet_copy_client.cc:783] T b26b6d19699047afaee57d6f2c887664 P d35c16dcb506426993811bb2519d7db3: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:14.622361 13305 tablet_copy_client.cc:647] T b26b6d19699047afaee57d6f2c887664 P d35c16dcb506426993811bb2519d7db3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:14.623361 13305 tablet_copy_client.cc:523] T b26b6d19699047afaee57d6f2c887664 P d35c16dcb506426993811bb2519d7db3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:14.624696 13305 tablet_bootstrap.cc:492] T b26b6d19699047afaee57d6f2c887664 P d35c16dcb506426993811bb2519d7db3: Bootstrap starting.
I20240501 14:10:14.629333 13305 tablet_bootstrap.cc:492] T b26b6d19699047afaee57d6f2c887664 P d35c16dcb506426993811bb2519d7db3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:10:14.629673 13305 tablet_bootstrap.cc:492] T b26b6d19699047afaee57d6f2c887664 P d35c16dcb506426993811bb2519d7db3: Bootstrap complete.
I20240501 14:10:14.629824 13305 ts_tablet_manager.cc:1367] T b26b6d19699047afaee57d6f2c887664 P d35c16dcb506426993811bb2519d7db3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240501 14:10:14.630337 13305 raft_consensus.cc:348] T b26b6d19699047afaee57d6f2c887664 P d35c16dcb506426993811bb2519d7db3 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:10:14.630596 13305 raft_consensus.cc:729] T b26b6d19699047afaee57d6f2c887664 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Initialized, Role: FOLLOWER
I20240501 14:10:14.630705 13305 consensus_queue.cc:260] T b26b6d19699047afaee57d6f2c887664 P d35c16dcb506426993811bb2519d7db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:14.631033 13305 ts_tablet_manager.cc:1398] T b26b6d19699047afaee57d6f2c887664 P d35c16dcb506426993811bb2519d7db3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:10:14.633675 32443 tablet_copy_service.cc:342] P a91029b90bf9415a89a2f70736759ad3: Request end of tablet copy session d35c16dcb506426993811bb2519d7db3-b26b6d19699047afaee57d6f2c887664 received from {username='slave'} at 127.22.252.130:47505
I20240501 14:10:14.633740 32443 tablet_copy_service.cc:433] P a91029b90bf9415a89a2f70736759ad3: ending tablet copy session d35c16dcb506426993811bb2519d7db3-b26b6d19699047afaee57d6f2c887664 on tablet b26b6d19699047afaee57d6f2c887664 with peer d35c16dcb506426993811bb2519d7db3
W20240501 14:10:14.680155 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:14.681526 13311 ts_tablet_manager.cc:902] T c90d65f39c3540f48c0d85b28130e76a P d1c217469ae4478ebd49e4ed9f7352d7: Initiating tablet copy from peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:10:14.681989 13311 tablet_copy_client.cc:308] T c90d65f39c3540f48c0d85b28130e76a P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.133:46265
I20240501 14:10:14.682411 32443 tablet_copy_service.cc:140] P a91029b90bf9415a89a2f70736759ad3: Received BeginTabletCopySession request for tablet c90d65f39c3540f48c0d85b28130e76a from peer d1c217469ae4478ebd49e4ed9f7352d7 ({username='slave'} at 127.22.252.132:53277)
I20240501 14:10:14.682505 32443 tablet_copy_service.cc:161] P a91029b90bf9415a89a2f70736759ad3: Beginning new tablet copy session on tablet c90d65f39c3540f48c0d85b28130e76a from peer d1c217469ae4478ebd49e4ed9f7352d7 at {username='slave'} at 127.22.252.132:53277: session id = d1c217469ae4478ebd49e4ed9f7352d7-c90d65f39c3540f48c0d85b28130e76a
I20240501 14:10:14.683509 32443 tablet_copy_source_session.cc:205] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:14.684432 13311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c90d65f39c3540f48c0d85b28130e76a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:14.686300 13311 tablet_copy_client.cc:783] T c90d65f39c3540f48c0d85b28130e76a P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:14.686480 13311 tablet_copy_client.cc:647] T c90d65f39c3540f48c0d85b28130e76a P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:14.687512 13311 tablet_copy_client.cc:523] T c90d65f39c3540f48c0d85b28130e76a P d1c217469ae4478ebd49e4ed9f7352d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:14.688596 13311 tablet_bootstrap.cc:492] T c90d65f39c3540f48c0d85b28130e76a P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap starting.
I20240501 14:10:14.691605 13311 tablet_bootstrap.cc:492] T c90d65f39c3540f48c0d85b28130e76a P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:14.691821 13311 tablet_bootstrap.cc:492] T c90d65f39c3540f48c0d85b28130e76a P d1c217469ae4478ebd49e4ed9f7352d7: Bootstrap complete.
I20240501 14:10:14.691996 13311 ts_tablet_manager.cc:1367] T c90d65f39c3540f48c0d85b28130e76a P d1c217469ae4478ebd49e4ed9f7352d7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:10:14.692445 13311 raft_consensus.cc:348] T c90d65f39c3540f48c0d85b28130e76a P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:14.692538 13311 raft_consensus.cc:729] T c90d65f39c3540f48c0d85b28130e76a P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Initialized, Role: FOLLOWER
I20240501 14:10:14.692659 13311 consensus_queue.cc:260] T c90d65f39c3540f48c0d85b28130e76a P d1c217469ae4478ebd49e4ed9f7352d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:14.692901 13311 ts_tablet_manager.cc:1398] T c90d65f39c3540f48c0d85b28130e76a P d1c217469ae4478ebd49e4ed9f7352d7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:10:14.693732 32443 tablet_copy_service.cc:342] P a91029b90bf9415a89a2f70736759ad3: Request end of tablet copy session d1c217469ae4478ebd49e4ed9f7352d7-c90d65f39c3540f48c0d85b28130e76a received from {username='slave'} at 127.22.252.132:53277
I20240501 14:10:14.693802 32443 tablet_copy_service.cc:433] P a91029b90bf9415a89a2f70736759ad3: ending tablet copy session d1c217469ae4478ebd49e4ed9f7352d7-c90d65f39c3540f48c0d85b28130e76a on tablet c90d65f39c3540f48c0d85b28130e76a with peer d1c217469ae4478ebd49e4ed9f7352d7
W20240501 14:10:14.744817 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:14.804427 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:14.868988 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:15.077491 30407 raft_consensus.cc:1203] T c90d65f39c3540f48c0d85b28130e76a P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:10:15.156692 30146 raft_consensus.cc:1203] T b26b6d19699047afaee57d6f2c887664 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:10:15.157258 11985 raft_consensus.cc:2932] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:15.158807 11985 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:37632 (request call id 0) took 1017 ms. Trace:
I20240501 14:10:15.158888 11985 rpcz_store.cc:276] 0501 14:10:14.141597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 14:10:14.141669 (+ 72us) service_pool.cc:225] Handling call
0501 14:10:14.142548 (+ 879us) log.cc:842] Serialized 496 byte log entry
0501 14:10:15.158792 (+1016244us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":55,"raft.queue_time_us":247,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":307,"spinlock_wait_cycles":12544,"thread_start_us":59,"threads_started":1,"wal-append.queue_time_us":123,"wal-append.run_cpu_time_us":296,"wal-append.run_wall_time_us":1000358}
I20240501 14:10:15.159811 30146 raft_consensus.cc:2932] T b26b6d19699047afaee57d6f2c887664 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:15.161232 29864 catalog_manager.cc:5484] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index 3 to 4, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:15.162714 347 raft_consensus.cc:2932] T b26b6d19699047afaee57d6f2c887664 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d35c16dcb506426993811bb2519d7db3 (127.22.252.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } }
I20240501 14:10:15.184799 30264 consensus_queue.cc:237] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:15.186321 30013 raft_consensus.cc:1261] T f2bc06e76e2a41429d5625ebf99e5e5c P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 14:10:15.186376 30212 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265): Couldn't send request to peer a91029b90bf9415a89a2f70736759ad3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2bc06e76e2a41429d5625ebf99e5e5c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:15.186796 12555 consensus_queue.cc:1035] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:10:15.187095 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:15.190740 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:15.346149 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:15.366569 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:15.373235 32621 consensus_peers.cc:485] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:15.377180 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:15.460878 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:15.539012 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:15.539793 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:15.555876 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:15.663632 13350 ts_tablet_manager.cc:902] T f2bc06e76e2a41429d5625ebf99e5e5c P a91029b90bf9415a89a2f70736759ad3: Initiating tablet copy from peer 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:15.664173 13350 tablet_copy_client.cc:308] T f2bc06e76e2a41429d5625ebf99e5e5c P a91029b90bf9415a89a2f70736759ad3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.252.131:43103
I20240501 14:10:15.664806 30297 tablet_copy_service.cc:140] P 1d0bb69611af4876a2b3b33c4766e13a: Received BeginTabletCopySession request for tablet f2bc06e76e2a41429d5625ebf99e5e5c from peer a91029b90bf9415a89a2f70736759ad3 ({username='slave'} at 127.22.252.133:60853)
I20240501 14:10:15.664897 30297 tablet_copy_service.cc:161] P 1d0bb69611af4876a2b3b33c4766e13a: Beginning new tablet copy session on tablet f2bc06e76e2a41429d5625ebf99e5e5c from peer a91029b90bf9415a89a2f70736759ad3 at {username='slave'} at 127.22.252.133:60853: session id = a91029b90bf9415a89a2f70736759ad3-f2bc06e76e2a41429d5625ebf99e5e5c
I20240501 14:10:15.666039 30297 tablet_copy_source_session.cc:205] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:10:15.666952 13350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2bc06e76e2a41429d5625ebf99e5e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:15.669005 13350 tablet_copy_client.cc:783] T f2bc06e76e2a41429d5625ebf99e5e5c P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 0 data blocks...
I20240501 14:10:15.669147 13350 tablet_copy_client.cc:647] T f2bc06e76e2a41429d5625ebf99e5e5c P a91029b90bf9415a89a2f70736759ad3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:15.670315 13350 tablet_copy_client.cc:523] T f2bc06e76e2a41429d5625ebf99e5e5c P a91029b90bf9415a89a2f70736759ad3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:15.671643 13350 tablet_bootstrap.cc:492] T f2bc06e76e2a41429d5625ebf99e5e5c P a91029b90bf9415a89a2f70736759ad3: Bootstrap starting.
W20240501 14:10:15.684873 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:15.692099 13350 tablet_bootstrap.cc:492] T f2bc06e76e2a41429d5625ebf99e5e5c P a91029b90bf9415a89a2f70736759ad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:10:15.692478 13350 tablet_bootstrap.cc:492] T f2bc06e76e2a41429d5625ebf99e5e5c P a91029b90bf9415a89a2f70736759ad3: Bootstrap complete.
I20240501 14:10:15.692636 13350 ts_tablet_manager.cc:1367] T f2bc06e76e2a41429d5625ebf99e5e5c P a91029b90bf9415a89a2f70736759ad3: Time spent bootstrapping tablet: real 0.021s user 0.020s sys 0.001s
I20240501 14:10:15.692996 13350 raft_consensus.cc:348] T f2bc06e76e2a41429d5625ebf99e5e5c P a91029b90bf9415a89a2f70736759ad3 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } }
I20240501 14:10:15.693214 13350 raft_consensus.cc:729] T f2bc06e76e2a41429d5625ebf99e5e5c P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91029b90bf9415a89a2f70736759ad3, State: Initialized, Role: FOLLOWER
I20240501 14:10:15.693356 13350 consensus_queue.cc:260] T f2bc06e76e2a41429d5625ebf99e5e5c P a91029b90bf9415a89a2f70736759ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:15.693653 13350 ts_tablet_manager.cc:1398] T f2bc06e76e2a41429d5625ebf99e5e5c P a91029b90bf9415a89a2f70736759ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:15.694296 30297 tablet_copy_service.cc:342] P 1d0bb69611af4876a2b3b33c4766e13a: Request end of tablet copy session a91029b90bf9415a89a2f70736759ad3-f2bc06e76e2a41429d5625ebf99e5e5c received from {username='slave'} at 127.22.252.133:60853
I20240501 14:10:15.694376 30297 tablet_copy_service.cc:433] P 1d0bb69611af4876a2b3b33c4766e13a: ending tablet copy session a91029b90bf9415a89a2f70736759ad3-f2bc06e76e2a41429d5625ebf99e5e5c on tablet f2bc06e76e2a41429d5625ebf99e5e5c with peer a91029b90bf9415a89a2f70736759ad3
W20240501 14:10:15.696156 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:15.746855 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:15.766806 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:15.854719 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:15.856760 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:15.920807 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:15.930732 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:15.961546 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:16.010746 32621 consensus_peers.cc:485] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:16.020680 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:16.028025 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:16.072209 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:16.084455 32416 raft_consensus.cc:1203] T f2bc06e76e2a41429d5625ebf99e5e5c P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 14:10:16.085160 11714 raft_consensus.cc:2932] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:16.085579 32416 raft_consensus.cc:2932] T f2bc06e76e2a41429d5625ebf99e5e5c P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:16.092023 29868 catalog_manager.cc:5484] T f2bc06e76e2a41429d5625ebf99e5e5c P a91029b90bf9415a89a2f70736759ad3 reported cstate change: config changed from index 5 to 6, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New cstate: current_term: 1 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
W20240501 14:10:16.094211 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:16.094632 30014 raft_consensus.cc:2932] T f2bc06e76e2a41429d5625ebf99e5e5c P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a91029b90bf9415a89a2f70736759ad3 (127.22.252.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
W20240501 14:10:16.137101 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:16.174505 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:16.177732 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:16.200171 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:16.241781 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:16.245386 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:16.249307 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:10:16.249548 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:16.250206 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:10:16.252959 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:16.258662 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:16.260651 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:16.266320 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:16.272387 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:16.312281 32621 consensus_peers.cc:485] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:16.350875 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:16.360559 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:16.407166 32621 consensus_peers.cc:485] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:16.409922 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:16.418782 13408 raft_consensus.cc:482] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:16.418907 13408 raft_consensus.cc:504] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:16.419399 13408 leader_election.cc:288] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:10:16.423772 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce3093e24347f3a4e121af452c6361" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:10:16.423935 30407 raft_consensus.cc:2454] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
W20240501 14:10:16.425189 32359 leader_election.cc:341] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Not found: Tablet not found: 89ce3093e24347f3a4e121af452c6361
W20240501 14:10:16.430047 32359 leader_election.cc:334] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:16.430121 32359 leader_election.cc:302] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3
I20240501 14:10:16.430424 13408 raft_consensus.cc:2726] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:16.436842 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:16.447355 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:16.449404 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:16.449694 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:10:16.453356 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:16.455086 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:16.458438 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:16.469535 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:16.473670 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:16.473877 32621 consensus_peers.cc:485] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:16.476820 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 14:10:16.496155 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:16.500730 32621 consensus_peers.cc:485] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:16.525415 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:16.532338 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:16.534584 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:16.567204 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:16.616693 30082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 14:10:16.617966 30344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 102 similar messages]
I20240501 14:10:16.655774 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:16.660274 32359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240501 14:10:16.662302 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:16.669673 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:16.671711 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:16.672209 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:16.677991 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:16.680420 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:10:16.692018 32753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 14:10:16.701032 29951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240501 14:10:16.701833 30213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 132 similar messages]
I20240501 14:10:16.714537 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:10:16.735304 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:16.747926 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:16.788218 32621 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:10:16.820300 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:16.856429 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:16.861723 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:16.862426 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:16.862802 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:16.864879 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:16.879321 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:16.881623 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:10:16.888965 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:16.904914 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:16.909937 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:16.939918 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:16.984135 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:17.062903 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:17.066089 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:17.069002 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:17.077025 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:17.080243 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:17.088368 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:17.091683 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:17.100778 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:10:17.143659 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:17.174783 32415 consensus_queue.cc:237] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } }
I20240501 14:10:17.176191 32684 raft_consensus.cc:1261] T c90d65f39c3540f48c0d85b28130e76a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:17.176432 30407 raft_consensus.cc:1261] T c90d65f39c3540f48c0d85b28130e76a P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Refusing update from remote peer a91029b90bf9415a89a2f70736759ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:17.176829 13323 consensus_queue.cc:1035] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:17.176990 13408 consensus_queue.cc:1035] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:17.179441 13323 raft_consensus.cc:2932] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:17.180101 32684 raft_consensus.cc:2932] T c90d65f39c3540f48c0d85b28130e76a P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:17.181578 30406 raft_consensus.cc:2932] T c90d65f39c3540f48c0d85b28130e76a P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:17.185041 29866 catalog_manager.cc:5484] T c90d65f39c3540f48c0d85b28130e76a P db935501e9104de5b15c8780681d6c38 reported cstate change: config changed from index 2 to 3, VOTER 93f2518fbadf4f9991cda93afacb363e (127.22.252.129) evicted. New cstate: current_term: 1 leader_uuid: "a91029b90bf9415a89a2f70736759ad3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } }
I20240501 14:10:17.188133 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet c90d65f39c3540f48c0d85b28130e76a with delete_type TABLET_DATA_TOMBSTONED (TS 93f2518fbadf4f9991cda93afacb363e not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46100
I20240501 14:10:17.188709 13658 tablet_replica.cc:330] T c90d65f39c3540f48c0d85b28130e76a P 93f2518fbadf4f9991cda93afacb363e: stopping tablet replica
I20240501 14:10:17.188915 13658 raft_consensus.cc:2229] T c90d65f39c3540f48c0d85b28130e76a P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:17.189087 13658 raft_consensus.cc:2258] T c90d65f39c3540f48c0d85b28130e76a P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:17.189587 13658 ts_tablet_manager.cc:1875] T c90d65f39c3540f48c0d85b28130e76a P 93f2518fbadf4f9991cda93afacb363e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:17.192339 13658 ts_tablet_manager.cc:1888] T c90d65f39c3540f48c0d85b28130e76a P 93f2518fbadf4f9991cda93afacb363e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:10:17.192418 13658 log.cc:1196] T c90d65f39c3540f48c0d85b28130e76a P 93f2518fbadf4f9991cda93afacb363e: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/wals/c90d65f39c3540f48c0d85b28130e76a
I20240501 14:10:17.193007 29859 catalog_manager.cc:4830] TS 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): tablet c90d65f39c3540f48c0d85b28130e76a (table rebalancer_extra_table_10 [id=c8e44abe11554acda26e71b8a11e2e0e]) successfully deleted
W20240501 14:10:17.232983 32621 consensus_peers.cc:485] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:17.269527 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:17.316561 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:17.317672 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:17.321786 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:17.330164 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:17.336406 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:10:17.343924 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:17.344691 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:17.360903 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:17.375721 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:17.424773 30013 consensus_queue.cc:237] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:17.425881 30145 raft_consensus.cc:1261] T f726c6a4929541c9b8ba7e29b757bb89 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:17.425947 32416 raft_consensus.cc:1261] T f726c6a4929541c9b8ba7e29b757bb89 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:10:17.426280 13356 consensus_queue.cc:1035] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:17.426437 13357 consensus_queue.cc:1035] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:10:17.428053 13356 raft_consensus.cc:2932] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:17.428480 30145 raft_consensus.cc:2932] T f726c6a4929541c9b8ba7e29b757bb89 P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:17.430756 32415 raft_consensus.cc:2932] T f726c6a4929541c9b8ba7e29b757bb89 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:17.432658 29870 catalog_manager.cc:5484] T f726c6a4929541c9b8ba7e29b757bb89 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: config changed from index 2 to 3, VOTER db935501e9104de5b15c8780681d6c38 (127.22.252.135) evicted. New cstate: current_term: 1 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:17.440075 32665 tablet_service.cc:1514] Processing DeleteTablet for tablet f726c6a4929541c9b8ba7e29b757bb89 with delete_type TABLET_DATA_TOMBSTONED (TS db935501e9104de5b15c8780681d6c38 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35078
I20240501 14:10:17.440613 13728 tablet_replica.cc:330] T f726c6a4929541c9b8ba7e29b757bb89 P db935501e9104de5b15c8780681d6c38: stopping tablet replica
I20240501 14:10:17.441097 13728 raft_consensus.cc:2229] T f726c6a4929541c9b8ba7e29b757bb89 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:17.441696 13728 raft_consensus.cc:2258] T f726c6a4929541c9b8ba7e29b757bb89 P db935501e9104de5b15c8780681d6c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:17.442247 13728 ts_tablet_manager.cc:1875] T f726c6a4929541c9b8ba7e29b757bb89 P db935501e9104de5b15c8780681d6c38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:10:17.446733 13728 ts_tablet_manager.cc:1888] T f726c6a4929541c9b8ba7e29b757bb89 P db935501e9104de5b15c8780681d6c38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 14:10:17.446810 13728 log.cc:1196] T f726c6a4929541c9b8ba7e29b757bb89 P db935501e9104de5b15c8780681d6c38: Deleting WAL directory at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/wals/f726c6a4929541c9b8ba7e29b757bb89
I20240501 14:10:17.447340 29857 catalog_manager.cc:4830] TS db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): tablet f726c6a4929541c9b8ba7e29b757bb89 (table rebalancer_extra_table_54 [id=b0f02750933049cf9e3457c96b5a5534]) successfully deleted
W20240501 14:10:17.518405 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:17.575621 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:17.581751 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:17.583458 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:17.585141 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:17.620846 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:17.621415 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:17.647140 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:10:17.681900 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:17.767627 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:17.815147 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:17.822085 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:17.825069 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:17.832645 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 14:10:17.841174 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:17.849861 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:10:17.855705 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:17.858531 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:10:17.859580 32621 consensus_peers.cc:485] T 21efaac3a3d643679c5abaaddb9e5a3a P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:17.863319 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:17.948855 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:17.969319 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:18.048348 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:18.052197 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:18.059916 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:18.060822 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:18.072396 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:18.073971 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:18.108099 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:18.133042 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:18.136451 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:18.137918 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:18.152801 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:18.182644 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:18.254395 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:18.267922 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:18.286900 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:18.289058 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:18.304330 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:18.306850 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:18.307579 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:18.315815 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:18.320425 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:10:18.342768 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:18.366937 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:18.385389 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:18.411121 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:18.468885 32621 consensus_peers.cc:485] T 6229c0c3e98c462b95f9bfe066367e87 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:10:18.507153 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:18.509773 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:18.512406 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:18.512748 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:18.525326 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:18.526400 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:18.535117 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:10:18.541899 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:18.542529 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:10:18.590260 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:18.605180 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:18.657131 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:18.663024 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:18.691601 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:18.699122 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:18.729000 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:10:18.730494 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:10:18.733290 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:10:18.735014 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:18.735404 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:18.740085 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 14:10:18.741209 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:18.742633 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:18.745968 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:18.756762 32621 consensus_peers.cc:485] T 5025e07cbfd543e3b4722c6c69e4a6b3 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:10:18.758090 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:18.800585 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:18.800683 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:18.886195 32621 consensus_peers.cc:485] T 0cccc8eec30141c1946b57d3be27000f P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:18.903285 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:18.908221 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:10:18.930320 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:18.932221 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:18.936941 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:18.937726 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:18.941202 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:18.955662 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:18.962064 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:10:18.982228 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:18.999572 13323 raft_consensus.cc:482] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:18.999706 13323 raft_consensus.cc:504] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:19.000186 13323 leader_election.cc:288] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
W20240501 14:10:19.001421 32359 leader_election.cc:341] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Not found: Tablet not found: 89ce3093e24347f3a4e121af452c6361
I20240501 14:10:19.001677 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce3093e24347f3a4e121af452c6361" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:10:19.001811 30407 raft_consensus.cc:2454] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
W20240501 14:10:19.001798 32359 leader_election.cc:334] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:19.001868 32359 leader_election.cc:302] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3
I20240501 14:10:19.002054 13323 raft_consensus.cc:2726] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:19.080751 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:19.108171 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:19.108801 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:19.129231 32621 consensus_peers.cc:485] T 36bdef51cf8a4256bedd1102d04c7aef P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:19.139338 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:10:19.140486 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:19.140997 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:10:19.142827 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:10:19.146160 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:19.147045 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:19.149916 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:19.165174 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:19.184473 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:19.203315 32621 consensus_peers.cc:485] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:19.232123 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:19.256661 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:19.256806 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:19.299911 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:19.371513 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:10:19.373804 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:19.377310 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:19.380609 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:10:19.384375 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:19.389492 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:19.399240 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:10:19.415563 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:10:19.416751 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:10:19.426230 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:10:19.513226 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:19.514822 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:19.530167 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:10:19.579836 327 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:19.584786 30125 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 14:10:19.593952 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:10:19.597785 30255 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:19.599773 29993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:10:19.605293 30386 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 14:10:19.634276 32665 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240501 14:10:19.638669 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:19.652757 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:10:19.698122 30013 consensus_queue.cc:237] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
W20240501 14:10:19.699591 29948 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3110aca9b1d643e4b07d0a869015b6f7. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:19.700083 32416 raft_consensus.cc:1261] T 3110aca9b1d643e4b07d0a869015b6f7 P a91029b90bf9415a89a2f70736759ad3 [term 2 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20240501 14:10:19.700448 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:19.700786 13357 consensus_queue.cc:1035] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 14:10:19.778077 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:19.813558 32621 consensus_peers.cc:485] T 65bc0e6ad1fa491e8e5ae225f86561f9 P db935501e9104de5b15c8780681d6c38 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:19.817478 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:19.830143 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:20.104439 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:20.181907 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:20.189292 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:20.215996 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:20.278049 14278 ts_tablet_manager.cc:902] T 3110aca9b1d643e4b07d0a869015b6f7 P db935501e9104de5b15c8780681d6c38: Initiating tablet copy from peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
F20240501 14:10:20.278424 14278 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-3110aca9b1d643e4b07d0a869015b6f7.queue_time_us, tablet-download-pool-3110aca9b1d643e4b07d0a869015b6f7.queue_time_us) (tablet-download-pool-3125c67fe40641c2bfd5dcac81e7237f.queue_time_us, tablet-download-pool-3125c67fe40641c2bfd5dcac81e7237f.queue_time_us) (tablet-download-pool-3125c67fe40641c2bfd5dcac81e7237f.run_cpu_time_us, tablet-download-pool-3125c67fe40641c2bfd5dcac81e7237f.run_cpu_time_us) (tablet-download-pool-3125c67fe40641c2bfd5dcac81e7237f.run_wall_time_us, tablet-download-pool-3125c67fe40641c2bfd5dcac81e7237f.run_wall_time_us) (tablet-download-pool-354658866eda4a2da6a2871fe0a2769d.queue_time_us, tablet-download-pool-354658866eda4a2da6a2871fe0a2769d.queue_time_us) (tablet-download-pool-354658866eda4a2da6a2871fe0a2769d.run_cpu_time_us, tablet-download-pool-354658866eda4a2da6a2871fe0a2769d.run_cpu_time_us) (tablet-download-pool-354658866eda4a2da6a2871fe0a2769d.run_wall_time_us, tablet-download-pool-354658866eda4a2da6a2871fe0a2769d.run_wall_time_us) (tablet-download-pool-3a04e958f3d049139ff74202611af725.queue_time_us, tablet-download-pool-3a04e958f3d049139ff74202611af725.queue_time_us) (tablet-download-pool-3a04e958f3d049139ff74202611af725.run_cpu_time_us, tablet-download-pool-3a04e958f3d049139ff74202611af725.run_cpu_time_us) (tablet-download-pool-3a04e958f3d049139ff74202611af725.run_wall_time_us, tablet-download-pool-3a04e958f3d049139ff74202611af725.run_wall_time_us) (tablet-download-pool-529f938305214c3ba33f61e06d1acd77.queue_time_us, tablet-download-pool-529f938305214c3ba33f61e06d1acd77.queue_time_us) (tablet-download-pool-529f938305214c3ba33f61e06d1acd77.run_cpu_time_us, tablet-download-pool-529f938305214c3ba33f61e06d1acd77.run_cpu_time_us) (tablet-download-pool-529f938305214c3ba33f61e06d1acd77.run_wall_time_us, tablet-download-pool-529f938305214c3ba33f61e06d1acd77.run_wall_time_us) (tablet-download-pool-75b67ccb9a334b25ae71b553a2a84a02.queue_time_us, tablet-download-pool-75b67ccb9a334b25ae71b553a2a84a02.queue_time_us) (tablet-download-pool-75b67ccb9a334b25ae71b553a2a84a02.run_cpu_time_us, tablet-download-pool-75b67ccb9a334b25ae71b553a2a84a02.run_cpu_time_us) (tablet-download-pool-75b67ccb9a334b25ae71b553a2a84a02.run_wall_time_us, tablet-download-pool-75b67ccb9a334b25ae71b553a2a84a02.run_wall_time_us) (tablet-download-pool-89d3e17f11a4460fa0d0a9a0b8342a66.queue_time_us, tablet-download-pool-89d3e17f11a4460fa0d0a9a0b8342a66.queue_time_us) (tablet-download-pool-89d3e17f11a4460fa0d0a9a0b8342a66.run_cpu_time_us, tablet-download-pool-89d3e17f11a4460fa0d0a9a0b8342a66.run_cpu_time_us) (tablet-download-pool-89d3e17f11a4460fa0d0a9a0b8342a66.run_wall_time_us, tablet-download-pool-89d3e17f11a4460fa0d0a9a0b8342a66.run_wall_time_us) (tablet-download-pool-af39c304693645269a6af59190e95cad.queue_time_us, tablet-download-pool-af39c304693645269a6af59190e95cad.queue_time_us) (tablet-download-pool-af39c304693645269a6af59190e95cad.run_cpu_time_us, tablet-download-pool-af39c304693645269a6af59190e95cad.run_cpu_time_us) (tablet-download-pool-af39c304693645269a6af59190e95cad.run_wall_time_us, tablet-download-pool-af39c304693645269a6af59190e95cad.run_wall_time_us) (tablet-download-pool-c362bd01a59b4bcf85eeb59713e58b64.queue_time_us, tablet-download-pool-c362bd01a59b4bcf85eeb59713e58b64.queue_time_us) (tablet-download-pool-c362bd01a59b4bcf85eeb59713e58b64.run_cpu_time_us, tablet-download-pool-c362bd01a59b4bcf85eeb59713e58b64.run_cpu_time_us) (tablet-download-pool-c362bd01a59b4bcf85eeb59713e58b64.run_wall_time_us, tablet-download-pool-c362bd01a59b4bcf85eeb59713e58b64.run_wall_time_us) (tablet-download-pool-dd2fd47ec6684cb28d2f4678d2e90e75.queue_time_us, tablet-download-pool-dd2fd47ec6684cb28d2f4678d2e90e75.queue_time_us) (tablet-download-pool-dd2fd47ec6684cb28d2f4678d2e90e75.run_cpu_time_us, tablet-download-pool-dd2fd47ec6684cb28d2f4678d2e90e75.run_cpu_time_us) (tablet-download-pool-dd2fd47ec6684cb28d2f4678d2e90e75.run_wall_time_us, tablet-download-pool-dd2fd47ec6684cb28d2f4678d2e90e75.run_wall_time_us) (tablet-download-pool-deee2ab875274d4eaeab84e698241ca8.queue_time_us, tablet-download-pool-deee2ab875274d4eaeab84e698241ca8.queue_time_us) (tablet-download-pool-deee2ab875274d4eaeab84e698241ca8.run_cpu_time_us, tablet-download-pool-deee2ab875274d4eaeab84e698241ca8.run_cpu_time_us) (tablet-download-pool-deee2ab875274d4eaeab84e698241ca8.run_wall_time_us, tablet-download-pool-deee2ab875274d4eaeab84e698241ca8.run_wall_time_us) (tablet-download-pool-f2541122e05f45c2843d06c5342c8d01.queue_time_us, tablet-download-pool-f2541122e05f45c2843d06c5342c8d01.queue_time_us) (tablet-download-pool-f2541122e05f45c2843d06c5342c8d01.run_cpu_time_us, tablet-download-pool-f2541122e05f45c2843d06c5342c8d01.run_cpu_time_us) (tablet-download-pool-f2541122e05f45c2843d06c5342c8d01.run_wall_time_us, tablet-download-pool-f2541122e05f45c2843d06c5342c8d01.run_wall_time_us) (tablet-download-pool-f740d894942a4bb584ca5688ef823d55.queue_time_us, tablet-download-pool-f740d894942a4bb584ca5688ef823d55.queue_time_us) (tablet-download-pool-f740d894942a4bb584ca5688ef823d55.run_cpu_time_us, tablet-download-pool-f740d894942a4bb584ca5688ef823d55.run_cpu_time_us) (tablet-download-pool-f740d894942a4bb584ca5688ef823d55.run_wall_time_us, tablet-download-pool-f740d894942a4bb584ca5688ef823d55.run_wall_time_us) (tablet-download-pool-fb466c8cb5a74897a884ac6012cc3734.queue_time_us, tablet-download-pool-fb466c8cb5a74897a884ac6012cc3734.queue_time_us) (tablet-download-pool-fb466c8cb5a74897a884ac6012cc3734.run_cpu_time_us, tablet-download-pool-fb466c8cb5a74897a884ac6012cc3734.run_cpu_time_us) (tablet-download-pool-fb466c8cb5a74897a884ac6012cc3734.run_wall_time_us, tablet-download-pool-fb466c8cb5a74897a884ac6012cc3734.run_wall_time_us) (tablet-download-pool-fc28b2028a9246b99e8887f03180d0c3.queue_time_us, tablet-download-pool-fc28b2028a9246b99e8887f03180d0c3.queue_time_us) (tablet-download-pool-fc28b2028a9246b99e8887f03180d0c3.run_cpu_time_us, tablet-download-pool-fc28b2028a9246b99e8887f03180d0c3.run_cpu_time_us) (tablet-download-pool-fc28b2028a9246b99e8887f03180d0c3.run_wall_time_us, tablet-download-pool-fc28b2028a9246b99e8887f03180d0c3.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f387d10edcd google::LogMessage::Fail() at ??:0
@ 0x7f387d112b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f387d10e7cc google::LogMessage::Flush() at ??:0
@ 0x7f387d10ff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f387e578044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f387e55cf98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f387e55aefc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f388110d77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f3881116a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f388119776f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f38811ab4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f3881196226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f38811a5c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x559bc8ed575c std::function<>::operator()() at ??:0
@ 0x7f387e55fe06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f387e5606ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f387e56203b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x559bc8ed575c std::function<>::operator()() at ??:0
@ 0x7f387e54f807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f387ed8d6db start_thread at ??:0
@ 0x7f387c2c571f clone at ??:0
W20240501 14:10:20.380492 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:20.407174 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:20.478797 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:20.500692 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:20.504091 29948 connection.cc:535] client connection to 127.22.252.135:41915 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:20.504174 30079 connection.cc:535] client connection to 127.22.252.135:41915 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:20.504168 30341 connection.cc:535] client connection to 127.22.252.135:41915 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:20.505116 32356 connection.cc:535] client connection to 127.22.252.135:41915 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:20.505174 30210 connection.cc:535] client connection to 127.22.252.135:41915 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:20.505908 11523 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:10:20.504659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.252.135:41915 (local address 127.22.252.129:50035)
0501 14:10:20.504888 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0501 14:10:20.504899 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 14:10:20.504991 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:10:20.505816 (+ 825us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":156}
W20240501 14:10:20.506197 11513 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:10:20.505602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.252.135:41915 (local address 127.22.252.131:57481)
0501 14:10:20.505864 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0501 14:10:20.505878 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 14:10:20.505955 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:10:20.506124 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":196}
W20240501 14:10:20.506232 11518 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:10:20.504905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.252.135:41915 (local address 127.22.252.130:42207)
0501 14:10:20.505027 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0501 14:10:20.505041 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 14:10:20.505109 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:10:20.506142 (+ 1033us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":49}
W20240501 14:10:20.506237 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.506316 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.506373 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.506507 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.506580 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.506633 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.506681 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.506731 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.506855 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.506873 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.506933 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:20.506973 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.506987 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.507010 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.507041 11525 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:10:20.504905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.252.135:41915 (local address 127.22.252.132:41603)
0501 14:10:20.505249 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0501 14:10:20.505261 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 14:10:20.505354 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:10:20.506433 (+ 1079us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":276}
W20240501 14:10:20.507189 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.507624 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.513216 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.521898 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.526060 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.529619 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:20.552104 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.563524 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.575188 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.581142 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.585402 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.597057 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.600049 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.610855 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.626556 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.638624 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.644960 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:20.669054 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.669992 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.675369 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.683897 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:20.689534 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.711911 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.712503 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.741742 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.752578 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:20.769695 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.772693 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.776098 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:20.777562 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:20.789444 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.795475 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.819236 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:20.821257 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:20.836274 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.845007 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:20.856741 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:20.958009 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:20.993774 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:21.032618 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:21.102672 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:21.134199 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:21.212090 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:21.216688 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:21.220381 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:21.254400 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:21.272675 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:21.277438 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:21.301525 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:10:21.310410 13323 raft_consensus.cc:482] T 59b8b59d67c64e2d828aaaa93647a5a4 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.310510 13323 raft_consensus.cc:504] T 59b8b59d67c64e2d828aaaa93647a5a4 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:21.310847 13323 leader_election.cc:288] T 59b8b59d67c64e2d828aaaa93647a5a4 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:10:21.311149 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59b8b59d67c64e2d828aaaa93647a5a4" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
W20240501 14:10:21.311717 32356 leader_election.cc:334] T 59b8b59d67c64e2d828aaaa93647a5a4 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.311780 32356 leader_election.cc:302] T 59b8b59d67c64e2d828aaaa93647a5a4 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3; no voters: d1c217469ae4478ebd49e4ed9f7352d7, db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.311889 13323 raft_consensus.cc:2726] T 59b8b59d67c64e2d828aaaa93647a5a4 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:21.341145 13323 raft_consensus.cc:482] T 270f2887aa3d43449695c61e2c2ae1d6 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.341248 13323 raft_consensus.cc:504] T 270f2887aa3d43449695c61e2c2ae1d6 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:21.341571 13323 leader_election.cc:288] T 270f2887aa3d43449695c61e2c2ae1d6 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:10:21.341856 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270f2887aa3d43449695c61e2c2ae1d6" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
W20240501 14:10:21.342510 32356 leader_election.cc:334] T 270f2887aa3d43449695c61e2c2ae1d6 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.342598 32356 leader_election.cc:302] T 270f2887aa3d43449695c61e2c2ae1d6 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3; no voters: 93f2518fbadf4f9991cda93afacb363e, db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.342729 13323 raft_consensus.cc:2726] T 270f2887aa3d43449695c61e2c2ae1d6 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:21.357509 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:21.377202 14299 raft_consensus.cc:482] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.377306 14299 raft_consensus.cc:504] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:21.377734 14299 leader_election.cc:288] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:10:21.378054 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8430bd8cb7549f0aa4ede9a3d1ffe6e" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d35c16dcb506426993811bb2519d7db3" is_pre_election: true
W20240501 14:10:21.378605 30210 leader_election.cc:334] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.378665 30210 leader_election.cc:302] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: d35c16dcb506426993811bb2519d7db3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.378801 14299 raft_consensus.cc:2726] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:21.383826 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:21.391371 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:21.393456 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:21.396196 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:10:21.408111 14299 raft_consensus.cc:482] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.408195 14299 raft_consensus.cc:504] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:21.408623 14299 leader_election.cc:288] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:10:21.408914 347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca8c2fa60c543dab6f30acce1958573" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
W20240501 14:10:21.409513 30210 leader_election.cc:334] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.409574 30210 leader_election.cc:302] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 9ecd907cf5634ac98f3f54f3dbec6849, db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.409698 14299 raft_consensus.cc:2726] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:21.417217 14266 raft_consensus.cc:482] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.417321 14266 raft_consensus.cc:504] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:21.417662 14266 leader_election.cc:288] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:21.418049 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e337eb723974834b07aac5eacf73232" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
W20240501 14:10:21.418553 32750 leader_election.cc:334] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.418622 32750 leader_election.cc:302] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849; no voters: 1d0bb69611af4876a2b3b33c4766e13a, db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.418748 14266 raft_consensus.cc:2726] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:21.480221 14003 raft_consensus.cc:482] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.480334 14003 raft_consensus.cc:504] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:21.480713 14003 leader_election.cc:288] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:10:21.480971 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6208e4b3d25941848b2ca533edd5a7b9" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
W20240501 14:10:21.481459 30341 leader_election.cc:334] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.481524 30341 leader_election.cc:302] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7; no voters: 93f2518fbadf4f9991cda93afacb363e, db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.481634 14003 raft_consensus.cc:2726] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:21.484225 14282 raft_consensus.cc:482] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.484321 14282 raft_consensus.cc:504] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:21.484419 14299 raft_consensus.cc:482] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.484493 14299 raft_consensus.cc:504] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:21.484755 14282 leader_election.cc:288] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:10:21.485253 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6208e4b3d25941848b2ca533edd5a7b9" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:10:21.485374 30407 raft_consensus.cc:2454] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:10:21.486351 29948 leader_election.cc:302] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:10:21.485323 14299 leader_election.cc:288] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:10:21.486723 14301 raft_consensus.cc:2781] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:21.486784 14301 raft_consensus.cc:482] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.486827 14301 raft_consensus.cc:3037] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:10:21.486964 30210 leader_election.cc:334] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.487862 14152 raft_consensus.cc:482] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.487854 14301 raft_consensus.cc:504] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:21.487933 14152 raft_consensus.cc:504] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:21.488160 14301 leader_election.cc:288] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:10:21.488322 14152 leader_election.cc:288] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:21.488941 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8430bd8cb7549f0aa4ede9a3d1ffe6e" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:10:21.489056 30276 raft_consensus.cc:2454] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 1.
W20240501 14:10:21.489166 30079 leader_election.cc:334] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:21.489188 29948 leader_election.cc:334] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.489328 30081 leader_election.cc:302] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters: db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.489339 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6208e4b3d25941848b2ca533edd5a7b9" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:10:21.489420 30407 raft_consensus.cc:3037] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:21.489567 14307 raft_consensus.cc:2781] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:21.489656 14307 raft_consensus.cc:482] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.489707 14307 raft_consensus.cc:3037] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:10:21.489799 29948 leader_election.cc:334] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.490258 30407 raft_consensus.cc:2454] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 2.
W20240501 14:10:21.490447 30213 leader_election.cc:334] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:21.490522 30213 leader_election.cc:302] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.490681 14306 raft_consensus.cc:2726] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:21.490696 14307 raft_consensus.cc:504] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:21.490813 29948 leader_election.cc:302] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters: db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.491008 14301 raft_consensus.cc:2781] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:21.491037 14307 leader_election.cc:288] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:21.491087 14301 raft_consensus.cc:686] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e [term 2 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:10:21.491204 14301 consensus_queue.cc:237] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:21.491542 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8430bd8cb7549f0aa4ede9a3d1ffe6e" candidate_uuid: "d35c16dcb506426993811bb2519d7db3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:10:21.491624 30276 raft_consensus.cc:3037] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:21.492594 30276 raft_consensus.cc:2454] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35c16dcb506426993811bb2519d7db3 in term 2.
W20240501 14:10:21.492609 30079 leader_election.cc:334] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.492823 30081 leader_election.cc:302] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d35c16dcb506426993811bb2519d7db3; no voters: db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.493690 29864 catalog_manager.cc:5484] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 1 to 2, leader changed from db935501e9104de5b15c8780681d6c38 (127.22.252.135) to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 2 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
I20240501 14:10:21.494416 14307 raft_consensus.cc:2781] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:21.494503 14307 raft_consensus.cc:686] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [term 2 LEADER]: Becoming Leader. State: Replica: d35c16dcb506426993811bb2519d7db3, State: Running, Role: LEADER
I20240501 14:10:21.494783 14307 consensus_queue.cc:237] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:21.496938 29869 catalog_manager.cc:5484] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 reported cstate change: term changed from 1 to 2, leader changed from db935501e9104de5b15c8780681d6c38 (127.22.252.135) to d35c16dcb506426993811bb2519d7db3 (127.22.252.130). New cstate: current_term: 2 leader_uuid: "d35c16dcb506426993811bb2519d7db3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
W20240501 14:10:21.500957 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:10:21.507334 14282 raft_consensus.cc:482] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.507431 14282 raft_consensus.cc:504] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:21.507722 14282 leader_election.cc:288] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:10:21.508170 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2cb58cf4854a238d6f1c4bfdc64ed7" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
W20240501 14:10:21.508594 29948 leader_election.cc:334] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.508749 29948 leader_election.cc:302] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: d1c217469ae4478ebd49e4ed9f7352d7, db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.508867 14301 raft_consensus.cc:2726] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:21.521238 14306 raft_consensus.cc:482] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.521363 14306 raft_consensus.cc:504] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:21.521699 14306 leader_election.cc:288] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:21.522596 30213 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:21.522671 30210 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.522744 30210 leader_election.cc:302] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.522867 14306 raft_consensus.cc:2726] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:10:21.529438 14266 raft_consensus.cc:482] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.529520 14266 raft_consensus.cc:504] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:21.529851 14266 leader_election.cc:288] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:21.530853 32753 leader_election.cc:334] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:21.530953 32750 leader_election.cc:334] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.531026 32750 leader_election.cc:302] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.531149 14266 raft_consensus.cc:2726] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:21.554628 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:10:21.565905 14003 raft_consensus.cc:482] T 8c579bea324147af88ea88ac30153f02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.565898 14306 raft_consensus.cc:482] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.565991 14003 raft_consensus.cc:504] T 8c579bea324147af88ea88ac30153f02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:21.565994 14306 raft_consensus.cc:504] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:21.566423 14003 leader_election.cc:288] T 8c579bea324147af88ea88ac30153f02 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:21.567152 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c579bea324147af88ea88ac30153f02" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
W20240501 14:10:21.567584 30210 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.567719 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025e07cbfd543e3b4722c6c69e4a6b3" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
W20240501 14:10:21.567876 30213 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:21.567946 30213 leader_election.cc:302] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.568030 14306 leader_election.cc:288] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:10:21.568092 14299 raft_consensus.cc:2726] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:21.569170 30341 leader_election.cc:334] T 8c579bea324147af88ea88ac30153f02 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.569233 30341 leader_election.cc:302] T 8c579bea324147af88ea88ac30153f02 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1c217469ae4478ebd49e4ed9f7352d7; no voters: 1d0bb69611af4876a2b3b33c4766e13a, db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.569365 14003 raft_consensus.cc:2726] T 8c579bea324147af88ea88ac30153f02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:21.610582 14301 raft_consensus.cc:482] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.610704 14301 raft_consensus.cc:504] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:21.611126 14301 leader_election.cc:288] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
W20240501 14:10:21.611326 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:10:21.616869 14303 raft_consensus.cc:482] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
W20240501 14:10:21.616883 29951 leader_election.cc:334] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:21.616971 14303 raft_consensus.cc:504] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:21.617347 14303 leader_election.cc:288] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
W20240501 14:10:21.617380 29948 leader_election.cc:334] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.617452 29948 leader_election.cc:302] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.617642 14301 raft_consensus.cc:2726] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:21.617929 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59b8b59d67c64e2d828aaaa93647a5a4" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:10:21.618064 32416 raft_consensus.cc:2454] T 59b8b59d67c64e2d828aaaa93647a5a4 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
W20240501 14:10:21.618134 30341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 115 similar messages]
I20240501 14:10:21.618549 30343 leader_election.cc:302] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:10:21.618703 14003 raft_consensus.cc:2781] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:21.618748 14003 raft_consensus.cc:482] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.618791 14003 raft_consensus.cc:3037] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:10:21.618854 30341 leader_election.cc:334] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.619799 14003 raft_consensus.cc:504] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:21.620111 14003 leader_election.cc:288] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:10:21.620682 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59b8b59d67c64e2d828aaaa93647a5a4" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:10:21.620785 32416 raft_consensus.cc:3037] T 59b8b59d67c64e2d828aaaa93647a5a4 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:10:21.621340 30079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240501 14:10:21.621769 32416 raft_consensus.cc:2454] T 59b8b59d67c64e2d828aaaa93647a5a4 P a91029b90bf9415a89a2f70736759ad3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 2.
I20240501 14:10:21.622372 30343 leader_election.cc:302] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:10:21.622514 14303 raft_consensus.cc:2781] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:21.622604 14303 raft_consensus.cc:686] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:10:21.622733 14303 consensus_queue.cc:237] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:21.626979 29869 catalog_manager.cc:5484] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 1 to 2, leader changed from db935501e9104de5b15c8780681d6c38 (127.22.252.135) to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 2 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
W20240501 14:10:21.628197 30341 leader_election.cc:334] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.631799 14301 raft_consensus.cc:482] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.631881 14301 raft_consensus.cc:504] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:21.632272 14301 leader_election.cc:288] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:10:21.632530 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270f2887aa3d43449695c61e2c2ae1d6" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:10:21.632630 32416 raft_consensus.cc:2454] T 270f2887aa3d43449695c61e2c2ae1d6 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 1.
I20240501 14:10:21.633008 29950 leader_election.cc:302] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, a91029b90bf9415a89a2f70736759ad3; no voters:
I20240501 14:10:21.633359 14282 raft_consensus.cc:2781] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:21.633409 14282 raft_consensus.cc:482] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting leader election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.633443 14282 raft_consensus.cc:3037] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:21.634435 14282 raft_consensus.cc:504] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:21.634740 14282 leader_election.cc:288] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 election: Requested vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:10:21.635067 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270f2887aa3d43449695c61e2c2ae1d6" candidate_uuid: "93f2518fbadf4f9991cda93afacb363e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:10:21.635161 32415 raft_consensus.cc:3037] T 270f2887aa3d43449695c61e2c2ae1d6 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:21.636058 32415 raft_consensus.cc:2454] T 270f2887aa3d43449695c61e2c2ae1d6 P a91029b90bf9415a89a2f70736759ad3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f2518fbadf4f9991cda93afacb363e in term 2.
W20240501 14:10:21.636250 29948 leader_election.cc:334] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:21.637311 29948 leader_election.cc:334] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.638967 29950 leader_election.cc:302] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, a91029b90bf9415a89a2f70736759ad3; no voters: db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.639168 14282 raft_consensus.cc:2781] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:21.639258 14282 raft_consensus.cc:686] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e [term 2 LEADER]: Becoming Leader. State: Replica: 93f2518fbadf4f9991cda93afacb363e, State: Running, Role: LEADER
I20240501 14:10:21.639400 14282 consensus_queue.cc:237] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:21.641705 29869 catalog_manager.cc:5484] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e reported cstate change: term changed from 1 to 2, leader changed from db935501e9104de5b15c8780681d6c38 (127.22.252.135) to 93f2518fbadf4f9991cda93afacb363e (127.22.252.129). New cstate: current_term: 2 leader_uuid: "93f2518fbadf4f9991cda93afacb363e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } }
I20240501 14:10:21.642243 14306 raft_consensus.cc:482] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.642356 14306 raft_consensus.cc:504] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:21.642714 14306 leader_election.cc:288] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:10:21.643013 347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e337eb723974834b07aac5eacf73232" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" is_pre_election: true
I20240501 14:10:21.643138 347 raft_consensus.cc:2454] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:10:21.643432 30212 leader_election.cc:302] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
W20240501 14:10:21.643734 30210 leader_election.cc:334] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.643913 14306 raft_consensus.cc:2781] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:21.643965 14306 raft_consensus.cc:482] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.644002 14306 raft_consensus.cc:3037] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:21.644924 14306 raft_consensus.cc:504] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:21.645212 14306 leader_election.cc:288] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136:45937)
I20240501 14:10:21.645501 347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e337eb723974834b07aac5eacf73232" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9ecd907cf5634ac98f3f54f3dbec6849"
I20240501 14:10:21.645598 347 raft_consensus.cc:3037] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:10:21.646095 30210 leader_election.cc:334] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.646454 347 raft_consensus.cc:2454] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 2.
I20240501 14:10:21.646731 30212 leader_election.cc:302] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 9ecd907cf5634ac98f3f54f3dbec6849; no voters: db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.646865 14306 raft_consensus.cc:2781] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:21.646946 14306 raft_consensus.cc:686] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:10:21.647060 14306 consensus_queue.cc:237] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:21.649518 29869 catalog_manager.cc:5484] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 1 to 2, leader changed from db935501e9104de5b15c8780681d6c38 (127.22.252.135) to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 2 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
W20240501 14:10:21.652060 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:21.657269 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:21.657965 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:10:21.676090 14266 raft_consensus.cc:482] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.676167 14266 raft_consensus.cc:504] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:10:21.676533 14266 leader_election.cc:288] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:21.677423 32753 leader_election.cc:334] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:21.677553 32750 leader_election.cc:334] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.677624 32750 leader_election.cc:302] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.677764 14266 raft_consensus.cc:2726] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:21.680766 14003 raft_consensus.cc:482] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.680859 14003 raft_consensus.cc:504] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:21.681222 14003 leader_election.cc:288] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:10:21.681545 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2cb58cf4854a238d6f1c4bfdc64ed7" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e" is_pre_election: true
I20240501 14:10:21.681674 30014 raft_consensus.cc:2454] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
I20240501 14:10:21.681885 30343 leader_election.cc:302] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters:
I20240501 14:10:21.682013 14003 raft_consensus.cc:2781] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:21.682060 14003 raft_consensus.cc:482] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.682085 14003 raft_consensus.cc:3037] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:10:21.682375 30341 leader_election.cc:334] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.683007 14003 raft_consensus.cc:504] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:21.683295 14003 leader_election.cc:288] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387)
I20240501 14:10:21.683604 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2cb58cf4854a238d6f1c4bfdc64ed7" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93f2518fbadf4f9991cda93afacb363e"
I20240501 14:10:21.683712 30014 raft_consensus.cc:3037] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:10:21.684181 30341 leader_election.cc:334] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.684640 30014 raft_consensus.cc:2454] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P 93f2518fbadf4f9991cda93afacb363e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 2.
I20240501 14:10:21.684896 30343 leader_election.cc:302] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e, d1c217469ae4478ebd49e4ed9f7352d7; no voters: db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.685051 14003 raft_consensus.cc:2781] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:21.685139 14003 raft_consensus.cc:686] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 LEADER]: Becoming Leader. State: Replica: d1c217469ae4478ebd49e4ed9f7352d7, State: Running, Role: LEADER
I20240501 14:10:21.685272 14003 consensus_queue.cc:237] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:21.687408 29869 catalog_manager.cc:5484] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 reported cstate change: term changed from 1 to 2, leader changed from db935501e9104de5b15c8780681d6c38 (127.22.252.135) to d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132). New cstate: current_term: 2 leader_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } } }
W20240501 14:10:21.703461 29951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 75 similar messages]
W20240501 14:10:21.704000 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:21.705742 30210 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 155 similar messages]
W20240501 14:10:21.715391 32750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240501 14:10:21.748301 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:10:21.765087 13323 raft_consensus.cc:482] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.765197 13323 raft_consensus.cc:504] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:21.765677 13323 leader_election.cc:288] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:21.765992 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025e07cbfd543e3b4722c6c69e4a6b3" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:10:21.766151 30276 raft_consensus.cc:2454] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
W20240501 14:10:21.766170 32356 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240501 14:10:21.766733 32356 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:21.766866 32359 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:21.766925 32359 leader_election.cc:302] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.767076 13323 raft_consensus.cc:2726] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:21.780251 13323 raft_consensus.cc:482] T 81750b74ac3f48e082e6e6d62ef8f066 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.780376 13323 raft_consensus.cc:504] T 81750b74ac3f48e082e6e6d62ef8f066 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:21.780735 13323 leader_election.cc:288] T 81750b74ac3f48e082e6e6d62ef8f066 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:21.781033 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81750b74ac3f48e082e6e6d62ef8f066" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
W20240501 14:10:21.781913 32356 leader_election.cc:334] T 81750b74ac3f48e082e6e6d62ef8f066 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.781980 32356 leader_election.cc:302] T 81750b74ac3f48e082e6e6d62ef8f066 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3; no voters: 1d0bb69611af4876a2b3b33c4766e13a, db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.782110 13648 raft_consensus.cc:2726] T 81750b74ac3f48e082e6e6d62ef8f066 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:21.784335 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:10:21.791355 14306 raft_consensus.cc:482] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.791433 14306 raft_consensus.cc:504] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:21.791724 14306 leader_election.cc:288] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:10:21.792024 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81750b74ac3f48e082e6e6d62ef8f066" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:10:21.792147 32415 raft_consensus.cc:2454] T 81750b74ac3f48e082e6e6d62ef8f066 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:10:21.792394 30212 leader_election.cc:302] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters:
W20240501 14:10:21.792585 30210 leader_election.cc:334] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.792739 14306 raft_consensus.cc:2781] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:21.792784 14306 raft_consensus.cc:482] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.792809 14306 raft_consensus.cc:3037] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:21.794044 14306 raft_consensus.cc:504] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:21.794380 14306 leader_election.cc:288] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:10:21.794662 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81750b74ac3f48e082e6e6d62ef8f066" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3"
I20240501 14:10:21.794768 32415 raft_consensus.cc:3037] T 81750b74ac3f48e082e6e6d62ef8f066 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:10:21.795296 30210 leader_election.cc:334] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.795773 32415 raft_consensus.cc:2454] T 81750b74ac3f48e082e6e6d62ef8f066 P a91029b90bf9415a89a2f70736759ad3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 2.
I20240501 14:10:21.796003 30212 leader_election.cc:302] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters: db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.796206 14306 raft_consensus.cc:2781] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:21.796319 14306 raft_consensus.cc:686] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:10:21.796624 14306 consensus_queue.cc:237] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } }
I20240501 14:10:21.798867 29869 catalog_manager.cc:5484] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 1 to 2, leader changed from db935501e9104de5b15c8780681d6c38 (127.22.252.135) to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 2 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } }
I20240501 14:10:21.802032 14301 raft_consensus.cc:482] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.802129 14301 raft_consensus.cc:504] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:21.802497 14301 leader_election.cc:288] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
I20240501 14:10:21.802641 14266 raft_consensus.cc:482] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.803017 14266 raft_consensus.cc:504] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
W20240501 14:10:21.803375 29951 leader_election.cc:334] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:21.803551 29948 leader_election.cc:334] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.803628 29948 leader_election.cc:302] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.803754 14301 raft_consensus.cc:2726] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:21.804126 14266 leader_election.cc:288] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
W20240501 14:10:21.804720 32750 leader_election.cc:334] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.804770 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca8c2fa60c543dab6f30acce1958573" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:10:21.804875 30276 raft_consensus.cc:2454] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 1.
I20240501 14:10:21.806303 14306 raft_consensus.cc:482] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.806398 14306 raft_consensus.cc:504] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:21.806720 14306 leader_election.cc:288] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:10:21.807161 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c579bea324147af88ea88ac30153f02" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:10:21.807294 30407 raft_consensus.cc:2454] T 8c579bea324147af88ea88ac30153f02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
W20240501 14:10:21.807472 30210 leader_election.cc:334] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.807595 30210 leader_election.cc:302] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters: db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.807739 14306 raft_consensus.cc:2781] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:21.807780 14306 raft_consensus.cc:482] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting leader election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.807819 14306 raft_consensus.cc:3037] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:21.808338 32752 leader_election.cc:302] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 9ecd907cf5634ac98f3f54f3dbec6849; no voters: db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.808485 14266 raft_consensus.cc:2781] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:10:21.808543 14266 raft_consensus.cc:482] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting leader election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:21.808584 14266 raft_consensus.cc:3037] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:21.809551 14266 raft_consensus.cc:504] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:21.809962 14266 leader_election.cc:288] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:21.810133 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca8c2fa60c543dab6f30acce1958573" candidate_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a"
I20240501 14:10:21.810248 30276 raft_consensus.cc:3037] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:21.810567 14306 raft_consensus.cc:504] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:21.811136 30276 raft_consensus.cc:2454] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecd907cf5634ac98f3f54f3dbec6849 in term 2.
I20240501 14:10:21.811604 32752 leader_election.cc:302] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, 9ecd907cf5634ac98f3f54f3dbec6849; no voters:
W20240501 14:10:21.811903 30210 leader_election.cc:334] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:21.812037 32750 leader_election.cc:334] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:21.812390 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:21.812423 14266 raft_consensus.cc:2781] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:21.812569 14266 raft_consensus.cc:686] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 LEADER]: Becoming Leader. State: Replica: 9ecd907cf5634ac98f3f54f3dbec6849, State: Running, Role: LEADER
I20240501 14:10:21.812691 14266 consensus_queue.cc:237] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:21.814378 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c579bea324147af88ea88ac30153f02" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7"
I20240501 14:10:21.814488 30407 raft_consensus.cc:3037] T 8c579bea324147af88ea88ac30153f02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:10:21.815142 29869 catalog_manager.cc:5484] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 reported cstate change: term changed from 1 to 2, leader changed from db935501e9104de5b15c8780681d6c38 (127.22.252.135) to 9ecd907cf5634ac98f3f54f3dbec6849 (127.22.252.136). New cstate: current_term: 2 leader_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:10:21.815506 30407 raft_consensus.cc:2454] T 8c579bea324147af88ea88ac30153f02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 2.
I20240501 14:10:21.816004 30210 leader_election.cc:302] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, d1c217469ae4478ebd49e4ed9f7352d7; no voters: db935501e9104de5b15c8780681d6c38
I20240501 14:10:21.816166 14302 raft_consensus.cc:2781] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:10:21.816249 14302 raft_consensus.cc:686] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 LEADER]: Becoming Leader. State: Replica: 1d0bb69611af4876a2b3b33c4766e13a, State: Running, Role: LEADER
I20240501 14:10:21.816371 14302 consensus_queue.cc:237] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:21.816941 14306 leader_election.cc:288] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 election: Requested vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357)
I20240501 14:10:21.818873 29869 catalog_manager.cc:5484] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a reported cstate change: term changed from 1 to 2, leader changed from db935501e9104de5b15c8780681d6c38 (127.22.252.135) to 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131). New cstate: current_term: 2 leader_uuid: "1d0bb69611af4876a2b3b33c4766e13a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } }
I20240501 14:10:21.876951 30407 raft_consensus.cc:1261] T 6208e4b3d25941848b2ca533edd5a7b9 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:10:21.877307 14301 consensus_queue.cc:1035] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:10:21.880772 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:21.881537 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:21.904724 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:21.923417 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:21.969311 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:21.983996 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:22.019963 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:10:22.040789 32415 raft_consensus.cc:1261] T 270f2887aa3d43449695c61e2c2ae1d6 P a91029b90bf9415a89a2f70736759ad3 [term 2 FOLLOWER]: Refusing update from remote peer 93f2518fbadf4f9991cda93afacb363e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:10:22.041194 14282 consensus_queue.cc:1035] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:10:22.041302 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:22.044706 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:22.048264 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:10:22.064947 347 raft_consensus.cc:1261] T 7e337eb723974834b07aac5eacf73232 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 2 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:10:22.065258 14302 consensus_queue.cc:1035] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:10:22.068059 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:22.068557 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:22.074793 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:22.078133 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:22.101650 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:22.109795 30276 raft_consensus.cc:1261] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Refusing update from remote peer d35c16dcb506426993811bb2519d7db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:10:22.110132 14152 consensus_queue.cc:1035] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:10:22.128684 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:22.167865 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:10:22.173336 30013 raft_consensus.cc:1261] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P 93f2518fbadf4f9991cda93afacb363e [term 2 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:10:22.174013 14303 consensus_queue.cc:1035] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:10:22.222821 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:22.244486 32415 raft_consensus.cc:1261] T 59b8b59d67c64e2d828aaaa93647a5a4 P a91029b90bf9415a89a2f70736759ad3 [term 2 FOLLOWER]: Refusing update from remote peer d1c217469ae4478ebd49e4ed9f7352d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:10:22.244844 14303 consensus_queue.cc:1035] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:10:22.251463 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:22.319257 30406 raft_consensus.cc:1261] T 8c579bea324147af88ea88ac30153f02 P d1c217469ae4478ebd49e4ed9f7352d7 [term 2 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:10:22.319677 14302 consensus_queue.cc:1035] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:10:22.322604 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:22.325598 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:22.345031 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:22.347357 32415 raft_consensus.cc:1261] T 81750b74ac3f48e082e6e6d62ef8f066 P a91029b90bf9415a89a2f70736759ad3 [term 2 FOLLOWER]: Refusing update from remote peer 1d0bb69611af4876a2b3b33c4766e13a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:10:22.347774 14302 consensus_queue.cc:1035] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:10:22.352751 30276 raft_consensus.cc:1261] T 0ca8c2fa60c543dab6f30acce1958573 P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Refusing update from remote peer 9ecd907cf5634ac98f3f54f3dbec6849: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:10:22.353157 14051 consensus_queue.cc:1035] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:10:22.410938 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:22.578399 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:22.606225 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:22.651409 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:22.652978 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:22.663756 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:22.715474 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:22.736445 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:22.757421 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:22.768930 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:22.825322 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:22.843886 13323 raft_consensus.cc:482] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:22.843999 13323 raft_consensus.cc:504] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:22.844422 13323 leader_election.cc:288] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:10:22.844696 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce3093e24347f3a4e121af452c6361" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:10:22.844848 30407 raft_consensus.cc:2454] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
W20240501 14:10:22.845448 32359 leader_election.cc:334] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:22.845626 32359 leader_election.cc:341] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Not found: Tablet not found: 89ce3093e24347f3a4e121af452c6361
I20240501 14:10:22.845696 32359 leader_election.cc:302] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3
I20240501 14:10:22.845850 13323 raft_consensus.cc:2726] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:22.846323 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:22.883236 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:22.908073 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:22.918915 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:22.956977 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:22.977348 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:22.994757 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.009281 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.014966 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.016358 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.035125 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.057338 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.059155 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:23.062891 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.063453 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.067780 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.077819 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.080504 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.093241 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.111408 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.112938 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:23.121069 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.121307 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.158686 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.158671 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.159785 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.163653 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.177384 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.185032 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.193719 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.206003 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.211922 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.213791 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.229100 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:23.234810 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:23.247545 14302 raft_consensus.cc:482] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:23.247642 14302 raft_consensus.cc:504] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:23.248001 14302 leader_election.cc:288] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:23.248860 30213 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:23.249238 30210 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:23.249318 30210 leader_election.cc:302] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:23.249446 14302 raft_consensus.cc:2726] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:23.251957 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:23.260903 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:23.314981 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.327098 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.327201 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:23.329202 14342 raft_consensus.cc:482] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:23.329313 14342 raft_consensus.cc:504] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:23.329672 14342 leader_election.cc:288] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:23.329835 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:23.330688 32753 leader_election.cc:334] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:23.330761 32750 leader_election.cc:334] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:23.330819 32750 leader_election.cc:302] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:23.330940 14342 raft_consensus.cc:2726] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:23.339295 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.361729 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.379429 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:23.391129 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:10:23.409466 14342 raft_consensus.cc:482] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:23.409590 14342 raft_consensus.cc:504] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:10:23.409986 14342 leader_election.cc:288] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:23.411036 32753 leader_election.cc:334] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:23.411118 32750 leader_election.cc:334] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:23.411195 32750 leader_election.cc:302] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:23.411489 14342 raft_consensus.cc:2726] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:23.422531 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.442215 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:23.449429 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.462517 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:10:23.479295 14282 raft_consensus.cc:482] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:23.479422 14282 raft_consensus.cc:504] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:23.479772 14282 leader_election.cc:288] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
W20240501 14:10:23.480931 29951 leader_election.cc:334] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:23.481176 29948 leader_election.cc:334] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:23.481248 29948 leader_election.cc:302] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:23.481382 14282 raft_consensus.cc:2726] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:23.481482 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:10:23.503623 13323 raft_consensus.cc:482] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:23.503738 13323 raft_consensus.cc:504] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:23.504554 13323 leader_election.cc:288] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:23.505589 30276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025e07cbfd543e3b4722c6c69e4a6b3" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
W20240501 14:10:23.505622 32356 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:23.505728 30276 raft_consensus.cc:2454] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
W20240501 14:10:23.506335 32359 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:23.506407 32359 leader_election.cc:302] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:23.506536 13323 raft_consensus.cc:2726] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:23.549566 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:23.565286 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:23.657539 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:23.657984 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:23.689180 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:10:23.712093 14302 raft_consensus.cc:482] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:23.712195 14302 raft_consensus.cc:504] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:23.712509 14302 leader_election.cc:288] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:23.713481 30213 leader_election.cc:334] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:23.713599 30210 leader_election.cc:334] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:23.713650 30210 leader_election.cc:302] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:23.713752 14302 raft_consensus.cc:2726] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:23.725600 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:23.732084 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:23.737097 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:23.742180 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:23.747520 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:23.785768 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:23.796900 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:23.853490 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:23.901810 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:23.916738 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:23.925410 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:23.925487 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:23.952461 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:23.979864 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:24.022781 14282 raft_consensus.cc:482] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:24.022948 14282 raft_consensus.cc:504] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:24.023392 14282 leader_election.cc:288] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:24.024442 29948 leader_election.cc:334] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:24.024533 29951 leader_election.cc:334] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:24.024621 29951 leader_election.cc:302] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:24.024776 14282 raft_consensus.cc:2726] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:24.129776 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:24.198884 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:24.204649 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:24.251590 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:24.265355 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:24.269311 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:24.292493 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:24.305145 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:24.334458 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:24.363355 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:24.366751 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:24.390116 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:24.395177 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:24.395793 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:24.410491 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:24.454978 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:24.482534 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:24.489630 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:24.520692 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:10:24.615531 13648 raft_consensus.cc:482] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:24.615694 13648 raft_consensus.cc:504] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:24.616143 13648 leader_election.cc:288] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:10:24.616535 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce3093e24347f3a4e121af452c6361" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:10:24.616662 30407 raft_consensus.cc:2454] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
W20240501 14:10:24.618579 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:24.622834 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:24.626874 32359 leader_election.cc:334] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:24.627238 32359 leader_election.cc:341] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Not found: Tablet not found: 89ce3093e24347f3a4e121af452c6361
I20240501 14:10:24.627307 32359 leader_election.cc:302] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3
I20240501 14:10:24.627424 13648 raft_consensus.cc:2726] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:24.648696 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:24.654079 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:24.658282 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:24.709278 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:24.756685 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:24.764086 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:24.771358 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:24.796926 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:24.891480 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:24.903605 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:25.073115 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.077250 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:25.103039 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:25.109627 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:25.127319 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:25.147738 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:25.226117 14346 raft_consensus.cc:482] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:25.226274 14346 raft_consensus.cc:504] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:25.226658 14346 leader_election.cc:288] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:25.227952 30210 leader_election.cc:334] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:25.228286 30213 leader_election.cc:334] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:25.228364 30213 leader_election.cc:302] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:25.228502 14346 raft_consensus.cc:2726] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:25.246479 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:25.275490 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.291106 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:25.307001 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.314705 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.324425 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.346735 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:25.349583 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.357560 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.371358 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.380545 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:25.415609 14352 raft_consensus.cc:482] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:25.415722 14352 raft_consensus.cc:504] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:25.416150 14352 leader_election.cc:288] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:25.417081 30213 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:25.417158 30210 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:25.417218 30210 leader_election.cc:302] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:25.417336 14352 raft_consensus.cc:2726] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:25.427188 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:25.427187 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:25.455296 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.493570 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.503149 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.507915 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.515765 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:25.520190 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.534003 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.536355 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.548811 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:25.559970 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:25.575178 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.581593 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.593297 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.602559 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.610539 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.633975 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.663772 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.664294 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.686034 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.686182 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.720664 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:25.740621 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:25.808676 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.817638 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.832437 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.839272 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.849097 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.850569 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.867302 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.869745 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:25.881382 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.889043 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.903311 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.908645 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:25.940790 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:25.949028 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:25.949458 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:10:25.978423 14051 raft_consensus.cc:482] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:25.978538 14051 raft_consensus.cc:504] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:25.978890 14051 leader_election.cc:288] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:25.979768 32753 leader_election.cc:334] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:25.979988 32750 leader_election.cc:334] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:25.980053 32750 leader_election.cc:302] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:25.983861 14342 raft_consensus.cc:2726] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:25.984355 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:10:25.985872 14346 raft_consensus.cc:482] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:25.985982 14346 raft_consensus.cc:504] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:25.986451 14346 leader_election.cc:288] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:10:25.986991 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025e07cbfd543e3b4722c6c69e4a6b3" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:10:25.987131 32415 raft_consensus.cc:2454] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
W20240501 14:10:25.987648 30213 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:25.992624 30210 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:25.992687 30210 leader_election.cc:302] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:25.992808 14346 raft_consensus.cc:2726] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:26.004212 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:26.023182 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:26.038829 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:26.041198 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:26.083726 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:26.139863 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:26.164086 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:26.215862 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:26.218658 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:26.228097 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:26.258445 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:26.300670 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:26.345453 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:26.352468 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:26.365818 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:10:26.391959 14282 raft_consensus.cc:482] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:26.392083 14282 raft_consensus.cc:504] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:26.392506 14282 leader_election.cc:288] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:26.393662 29951 leader_election.cc:334] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:26.393931 29948 leader_election.cc:334] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:26.394006 29948 leader_election.cc:302] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:26.394147 14282 raft_consensus.cc:2726] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:26.394284 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:26.435020 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:26.440402 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:26.473618 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:10:26.505151 14282 raft_consensus.cc:482] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:26.505267 14282 raft_consensus.cc:504] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:26.505589 14282 leader_election.cc:288] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
W20240501 14:10:26.506657 29951 leader_election.cc:334] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:26.506742 29948 leader_election.cc:334] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:26.506803 29948 leader_election.cc:302] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:26.506937 14282 raft_consensus.cc:2726] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:26.527997 14051 raft_consensus.cc:482] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:26.528105 14051 raft_consensus.cc:504] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:10:26.528555 14051 leader_election.cc:288] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:26.529503 32750 leader_election.cc:334] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:26.529574 32753 leader_election.cc:334] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:26.529644 32750 leader_election.cc:302] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:26.529764 14051 raft_consensus.cc:2726] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:26.569314 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:26.586833 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:26.603947 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:26.624490 30344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 182 similar messages]
W20240501 14:10:26.665616 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:26.702813 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:26.710430 29951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 161 similar messages]
W20240501 14:10:26.712710 30210 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 236 similar messages]
W20240501 14:10:26.714408 30079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:10:26.721244 32753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 188 similar messages]
W20240501 14:10:26.735600 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:26.765293 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:26.775887 32359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 147 similar messages]
W20240501 14:10:26.782944 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:26.784615 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:26.786526 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:26.791324 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:26.798712 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:26.834806 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:26.878156 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:26.904947 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:26.929087 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:26.945179 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:26.955377 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:26.960794 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:26.984783 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:27.005218 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:27.060045 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:27.097772 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:27.129976 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:27.132478 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:27.143054 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:27.224073 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:27.229202 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:27.287830 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:27.318162 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:27.329732 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:27.340675 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:27.347195 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:27.366997 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:27.586228 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:27.605520 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:27.611481 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:27.630561 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:27.673856 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:27.711138 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:27.711447 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:27.725944 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:27.730872 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:27.744050 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:27.744827 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:27.803903 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:27.820168 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:27.861757 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:27.864646 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:27.868317 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:27.896514 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:27.913642 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:27.958861 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:27.976490 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.001895 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.019948 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:28.035207 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.035693 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.050832 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.059216 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:28.083644 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.105799 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.116578 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.146004 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.158942 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.164793 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.166051 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.167624 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.175151 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.175964 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:28.184490 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.185398 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.185969 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.194762 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.197768 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.203068 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.223282 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.232034 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:28.246053 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:28.253108 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:28.273103 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:28.308183 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.312551 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.356150 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.368461 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:28.385444 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.402727 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:28.406360 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:28.437422 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:10:28.453161 14365 raft_consensus.cc:482] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:28.453280 14365 raft_consensus.cc:504] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
W20240501 14:10:28.453500 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:10:28.453759 14365 leader_election.cc:288] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:28.454129 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025e07cbfd543e3b4722c6c69e4a6b3" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:10:28.454267 30277 raft_consensus.cc:2454] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
W20240501 14:10:28.454762 32356 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:28.455090 32359 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:28.455157 32359 leader_election.cc:302] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:28.455279 14365 raft_consensus.cc:2726] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:28.461241 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:28.508566 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:10:28.545074 14364 raft_consensus.cc:482] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:28.545182 14364 raft_consensus.cc:504] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:28.545558 14364 leader_election.cc:288] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:28.546411 32753 leader_election.cc:334] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:28.546489 32750 leader_election.cc:334] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:28.546547 32750 leader_election.cc:302] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:28.546661 14364 raft_consensus.cc:2726] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:28.558446 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.561834 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.603336 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:28.617851 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:28.619714 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:28.681667 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:10:28.698310 14364 raft_consensus.cc:482] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:28.698415 14364 raft_consensus.cc:504] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:10:28.698802 14364 leader_election.cc:288] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:28.699684 32753 leader_election.cc:334] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:28.699812 32750 leader_election.cc:334] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:28.699893 32750 leader_election.cc:302] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:28.700030 14364 raft_consensus.cc:2726] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:28.702088 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:28.709926 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:10:28.726285 14371 raft_consensus.cc:482] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:28.726403 14371 raft_consensus.cc:504] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:28.726764 14371 leader_election.cc:288] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:28.727706 29951 leader_election.cc:334] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:28.727842 29948 leader_election.cc:334] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:28.727914 29948 leader_election.cc:302] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:28.728044 14371 raft_consensus.cc:2726] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:28.808413 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:28.816298 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:28.823626 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:28.877382 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:28.888592 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:28.916406 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:28.922536 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:28.961043 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:28.963071 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:29.073053 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:29.081130 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:29.106629 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:29.107509 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:29.120647 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:29.182904 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:29.189786 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:10:29.210306 14352 raft_consensus.cc:482] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:29.210454 14352 raft_consensus.cc:504] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:29.210793 14352 leader_election.cc:288] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:29.211699 30210 leader_election.cc:334] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:29.211812 30213 leader_election.cc:334] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:29.211882 30213 leader_election.cc:302] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:29.211997 14352 raft_consensus.cc:2726] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:29.224749 14352 raft_consensus.cc:482] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:29.224831 14352 raft_consensus.cc:504] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:29.225212 14352 leader_election.cc:288] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:29.226153 30213 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:29.226248 30210 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:29.226325 30210 leader_election.cc:302] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:29.226452 14352 raft_consensus.cc:2726] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:29.241531 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:29.268389 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:29.268924 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:29.279160 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:29.301155 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:29.334175 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:29.336272 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:29.349277 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:29.350845 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:29.370540 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:29.376080 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:29.377593 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:29.385262 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:29.435231 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:29.441447 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:29.495857 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:29.550163 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:29.623482 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:29.709831 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:29.718425 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:29.721908 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:29.827375 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:10:29.875983 14282 raft_consensus.cc:482] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:29.876176 14282 raft_consensus.cc:504] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:29.876919 14282 leader_election.cc:288] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
W20240501 14:10:29.877482 29951 leader_election.cc:334] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:29.877597 29948 leader_election.cc:334] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:29.877652 29948 leader_election.cc:302] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:29.877784 14282 raft_consensus.cc:2726] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:29.890028 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:29.891162 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:29.903215 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:29.957290 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:30.018131 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:30.023489 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:30.123085 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:30.136969 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.190479 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.195760 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:30.216675 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.218976 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:30.230223 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:30.257592 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:30.301340 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:30.301316 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.309926 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.324695 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:30.328531 14379 raft_consensus.cc:482] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:30.328639 14379 raft_consensus.cc:504] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:30.329011 14379 leader_election.cc:288] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:30.330134 29951 leader_election.cc:334] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:30.330298 29948 leader_election.cc:334] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:30.330423 29948 leader_election.cc:302] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:30.330729 14379 raft_consensus.cc:2726] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:30.332866 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.377547 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.382112 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:30.387387 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.416028 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.434021 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:30.440996 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.449818 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.461922 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.464072 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.471058 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:30.480732 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.484483 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.500270 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.523344 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:30.532857 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.537472 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:30.540594 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:30.551615 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.613864 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.621637 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.624742 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.645143 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.648612 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.663031 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.678089 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:30.682984 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.693713 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:30.714625 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:30.748436 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.751814 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.755117 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.805249 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.850828 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:30.872161 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.876883 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.888764 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.892717 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.921591 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:30.964289 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:30.972055 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:30.992121 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:30.996990 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:31.049333 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:10:31.067075 14385 raft_consensus.cc:482] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:31.067200 14385 raft_consensus.cc:504] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:31.067752 14385 leader_election.cc:288] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:10:31.067943 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce3093e24347f3a4e121af452c6361" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:10:31.068117 32415 raft_consensus.cc:2454] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
W20240501 14:10:31.068707 30344 leader_election.cc:341] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Not found: Tablet not found: 89ce3093e24347f3a4e121af452c6361
W20240501 14:10:31.068825 30344 leader_election.cc:334] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:31.068876 30344 leader_election.cc:302] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3
I20240501 14:10:31.069008 14385 raft_consensus.cc:2726] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:31.137969 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:31.148648 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:31.198844 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:31.229498 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:31.230623 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:31.284718 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:31.304809 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:10:31.313653 14376 raft_consensus.cc:482] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:31.313758 14376 raft_consensus.cc:504] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:31.314116 14376 leader_election.cc:288] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:31.315207 30213 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:31.315300 30210 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:31.315382 30210 leader_election.cc:302] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:31.315526 14376 raft_consensus.cc:2726] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:31.333109 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:31.364063 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:31.375399 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:31.380321 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:31.380908 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:31.393550 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:31.424023 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:31.441511 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:31.472040 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:31.492691 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:31.514271 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:31.555282 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:31.594058 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:31.606606 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:31.631896 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:31.658975 30341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 181 similar messages]
W20240501 14:10:31.675370 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:31.679443 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:31.685253 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:31.705698 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:31.722226 30079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240501 14:10:31.723733 32753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 184 similar messages]
W20240501 14:10:31.725325 29951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 155 similar messages]
W20240501 14:10:31.726303 30210 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 226 similar messages]
W20240501 14:10:31.734359 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:31.734459 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:31.735723 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:31.742352 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:31.759100 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:31.786844 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:31.809964 32359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240501 14:10:31.815387 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:31.816951 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:31.854640 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:31.860002 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:31.981891 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:31.996459 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:32.025379 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:32.030351 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:32.035296 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:32.041375 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:32.044754 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:32.059711 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:32.209645 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:10:32.321415 14391 raft_consensus.cc:482] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:32.321523 14391 raft_consensus.cc:504] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:32.321828 14391 leader_election.cc:288] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
W20240501 14:10:32.322675 29951 leader_election.cc:334] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:32.322800 29948 leader_election.cc:334] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:32.322861 29948 leader_election.cc:302] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:32.323010 14391 raft_consensus.cc:2726] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:32.334851 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:10:32.337056 14364 raft_consensus.cc:482] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:32.337150 14364 raft_consensus.cc:504] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:32.337541 14364 leader_election.cc:288] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:32.338791 32753 leader_election.cc:334] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:32.339206 32750 leader_election.cc:334] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:32.339273 32750 leader_election.cc:302] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:32.339411 14364 raft_consensus.cc:2726] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:10:32.345882 14364 raft_consensus.cc:482] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:32.345983 14364 raft_consensus.cc:504] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:10:32.346349 14364 leader_election.cc:288] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:32.347219 32753 leader_election.cc:334] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:32.347309 32750 leader_election.cc:334] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:32.347383 32750 leader_election.cc:302] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:32.347502 14364 raft_consensus.cc:2726] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:32.408470 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:32.418725 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:32.422358 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:32.496282 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:32.532742 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:32.565680 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:32.593844 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:32.623410 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:32.661059 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:32.684597 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:32.694185 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:32.695809 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:10:32.711170 14392 raft_consensus.cc:482] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:32.711333 14392 raft_consensus.cc:504] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:32.711916 14392 leader_election.cc:288] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:32.712260 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025e07cbfd543e3b4722c6c69e4a6b3" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:10:32.712415 30277 raft_consensus.cc:2454] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
W20240501 14:10:32.713079 32359 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:32.713191 32356 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:32.713281 32356 leader_election.cc:302] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:32.713411 14392 raft_consensus.cc:2726] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:32.745055 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:32.763576 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:32.767957 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:32.805900 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:32.808197 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:32.823848 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:32.833318 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:32.839375 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:32.841820 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:32.860946 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:32.879932 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:32.928050 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:32.952080 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:32.956677 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:32.963632 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:32.967734 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:32.974256 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:32.979301 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:32.981472 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:32.985319 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.009577 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.019054 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.019376 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.027500 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.071830 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.073882 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.087760 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:33.095604 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:33.108073 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.135668 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.189504 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.193418 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.228080 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:33.240588 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.271670 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.283035 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:33.299690 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.305886 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.339504 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.350211 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:33.368872 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:33.374370 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.459179 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.469112 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.484534 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.486984 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.490635 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:33.490865 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:33.492630 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.528429 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:33.541286 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:33.559382 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:33.611368 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:33.621554 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:33.668794 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.710392 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:33.725239 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:33.732256 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:33.739864 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.775439 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:33.789206 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:33.839951 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:33.844687 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:33.848071 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.903681 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:33.919744 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:33.939234 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:33.961783 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:34.041509 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:34.050043 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:34.065255 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:34.072212 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:10:34.074409 14374 raft_consensus.cc:482] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:34.074502 14374 raft_consensus.cc:504] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:10:34.074883 14374 leader_election.cc:288] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:34.075740 32753 leader_election.cc:334] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:34.076141 32750 leader_election.cc:334] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:34.076208 32750 leader_election.cc:302] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:34.076354 14374 raft_consensus.cc:2726] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:34.085507 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:34.093395 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:34.121202 14391 raft_consensus.cc:482] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:34.121335 14391 raft_consensus.cc:504] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:34.121760 14391 leader_election.cc:288] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
W20240501 14:10:34.122710 29948 leader_election.cc:334] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:34.122800 29951 leader_election.cc:334] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:34.122897 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:10:34.122906 29951 leader_election.cc:302] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:34.123060 14391 raft_consensus.cc:2726] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:34.143954 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:34.182650 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:34.187417 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:34.188370 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:34.195760 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:34.201375 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:34.268437 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:34.325846 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:34.335872 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:34.393318 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:34.500329 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:34.508196 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:34.513999 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:34.526723 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:34.567229 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:34.573290 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:10:34.587402 14369 raft_consensus.cc:482] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:34.587507 14369 raft_consensus.cc:504] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:34.588038 14369 leader_election.cc:288] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:10:34.588261 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce3093e24347f3a4e121af452c6361" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:10:34.588575 30407 raft_consensus.cc:2454] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
W20240501 14:10:34.589033 32359 leader_election.cc:334] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:34.589205 32359 leader_election.cc:341] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Not found: Tablet not found: 89ce3093e24347f3a4e121af452c6361
I20240501 14:10:34.589267 32359 leader_election.cc:302] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3
I20240501 14:10:34.589483 14369 raft_consensus.cc:2726] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:34.590051 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:34.621762 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:34.634073 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:34.662117 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:34.712651 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:34.750432 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:34.797267 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:34.890944 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:34.926575 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:34.966235 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:35.068383 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:35.150259 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:35.189250 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:35.194842 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:35.194823 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.199590 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:35.205428 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.257329 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:35.278692 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:35.298501 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:35.325109 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:35.334089 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.368670 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.370587 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.378217 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.388914 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:35.392438 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.408524 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:35.411170 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.446022 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.450381 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.460569 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.465801 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.477941 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.478402 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.483064 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.526439 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.527974 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:35.529834 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.543546 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.558642 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.596812 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:35.622026 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.626802 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.632191 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:35.635444 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.635588 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.636214 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:35.690711 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:35.710757 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.712177 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.726902 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.744107 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.770673 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:35.771279 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:35.798794 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:35.807000 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.846493 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.860551 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.862259 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:10:35.877655 14396 raft_consensus.cc:482] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:35.877739 14396 raft_consensus.cc:504] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:35.878100 14396 leader_election.cc:288] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:35.879055 30213 leader_election.cc:334] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:35.879173 30210 leader_election.cc:334] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:35.879249 30210 leader_election.cc:302] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:35.879386 14396 raft_consensus.cc:2726] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:35.881426 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.891523 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.893795 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.898391 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.929986 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:35.932736 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:35.956934 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:10:35.962349 14412 raft_consensus.cc:482] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:35.962437 14412 raft_consensus.cc:504] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:35.962782 14412 leader_election.cc:288] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:35.963547 29951 leader_election.cc:334] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:35.963629 29948 leader_election.cc:334] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:35.963717 29948 leader_election.cc:302] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:35.963829 14412 raft_consensus.cc:2726] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:36.049366 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:36.061880 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:36.067710 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:36.083038 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:36.092567 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:36.154384 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:36.201563 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:36.244738 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:36.270958 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:36.320832 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:36.340040 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:36.405323 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:36.408574 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:10:36.429726 14374 raft_consensus.cc:482] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:36.430230 14374 raft_consensus.cc:504] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:36.430658 14374 leader_election.cc:288] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:36.431254 32753 leader_election.cc:334] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:36.431818 32750 leader_election.cc:334] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:36.431883 32750 leader_election.cc:302] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:36.432072 14374 raft_consensus.cc:2726] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:36.434749 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:36.454025 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:36.456440 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:36.481963 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:36.514142 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:36.570135 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:36.572825 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:36.575105 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:36.584718 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:36.600541 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:36.633105 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:36.675014 30344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 177 similar messages]
W20240501 14:10:36.680581 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:36.685127 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:36.690438 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:36.733716 30213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 221 similar messages]
W20240501 14:10:36.743314 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:36.754244 29948 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 156 similar messages]
W20240501 14:10:36.754773 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:36.776908 32750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 189 similar messages]
W20240501 14:10:36.781430 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:36.787614 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:36.787652 30082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 138 similar messages]
I20240501 14:10:36.814855 14403 raft_consensus.cc:482] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:36.814946 14403 raft_consensus.cc:504] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:36.815397 14403 leader_election.cc:288] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:10:36.815886 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025e07cbfd543e3b4722c6c69e4a6b3" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:10:36.816035 32415 raft_consensus.cc:2454] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
W20240501 14:10:36.816280 30210 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:36.816797 30213 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:36.816852 30213 leader_election.cc:302] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:36.816968 14403 raft_consensus.cc:2726] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:36.820484 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:36.821650 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:36.829505 32356 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240501 14:10:36.892222 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:36.907881 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:37.017139 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:37.022135 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:37.045231 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:37.087648 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:37.102300 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:37.108412 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:37.126274 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:37.130049 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:37.142977 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:37.151520 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:37.190848 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:37.240937 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:37.243539 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:37.260164 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:37.266420 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:37.342170 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:37.366000 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:37.379460 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:37.440412 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:37.496819 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:10:37.499140 14415 raft_consensus.cc:482] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:37.499290 14415 raft_consensus.cc:504] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:37.499672 14415 leader_election.cc:288] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
W20240501 14:10:37.500792 29948 leader_election.cc:334] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:37.501006 29951 leader_election.cc:334] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:37.501117 29951 leader_election.cc:302] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:37.501266 14415 raft_consensus.cc:2726] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:37.576625 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:37.582329 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:37.655795 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:10:37.677745 14420 raft_consensus.cc:482] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:37.677860 14420 raft_consensus.cc:504] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:37.678208 14420 leader_election.cc:288] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:37.679075 30213 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:37.679256 30210 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:37.679324 30210 leader_election.cc:302] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:37.679467 14420 raft_consensus.cc:2726] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:37.715768 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:37.717027 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:37.730262 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:37.745811 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:37.755848 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:37.764209 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:37.768123 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:37.778409 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:37.800060 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:37.818552 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:37.825692 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:37.849227 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:37.866886 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:37.900656 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:37.911610 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:37.912654 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:37.921088 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:37.951126 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:37.979261 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:37.985610 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.025492 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:38.037815 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.044943 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.066855 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.080250 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:38.087415 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.088081 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.109513 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:38.126001 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.180856 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.209318 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:38.212872 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.245586 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.262670 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:38.282575 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.310385 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.348661 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:38.359215 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:38.372076 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:38.384212 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.384357 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.385874 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.398829 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.420598 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:38.438841 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.446873 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.460057 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.510396 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:38.526525 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.576294 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.599732 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.640803 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:38.649410 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:10:38.672989 14403 raft_consensus.cc:482] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:38.673116 14403 raft_consensus.cc:504] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:38.673609 14403 leader_election.cc:288] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:10:38.674129 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025e07cbfd543e3b4722c6c69e4a6b3" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:10:38.674264 32415 raft_consensus.cc:2454] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
W20240501 14:10:38.674474 30210 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:38.675151 30213 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:38.675211 30213 leader_election.cc:302] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:38.675334 14403 raft_consensus.cc:2726] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:38.693974 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:38.694964 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.755159 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:38.796784 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:38.817559 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:38.863319 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:38.866554 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:38.869565 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:38.879307 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:38.887495 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:38.905568 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:38.932962 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:38.943840 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:38.963178 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:38.991557 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:39.028040 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:39.030910 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:39.060101 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:39.076540 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:39.083313 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:39.124300 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:39.207302 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:39.212303 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:39.252624 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:39.266168 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:39.337755 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:39.350881 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:39.355931 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:39.380365 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:39.403306 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:39.475172 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:39.501977 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:39.549994 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:39.573730 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:39.587191 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:39.587888 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:39.595374 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:39.598099 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:39.608471 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:39.631254 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:39.664633 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:39.699313 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:39.714983 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:39.725230 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:39.741963 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:39.765390 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:39.777514 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:39.782938 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:39.847745 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:39.850987 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:39.885980 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:39.904227 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:39.980281 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:40.170275 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.184357 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.187178 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:10:40.189787 14403 raft_consensus.cc:482] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:40.189869 14403 raft_consensus.cc:504] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:40.190284 14403 leader_election.cc:288] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:10:40.190651 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025e07cbfd543e3b4722c6c69e4a6b3" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:10:40.190784 32416 raft_consensus.cc:2454] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
W20240501 14:10:40.191165 30210 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:40.191314 30213 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:40.191370 30213 leader_election.cc:302] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:40.191478 14403 raft_consensus.cc:2726] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:40.193449 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:40.197193 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:40.236356 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:40.261591 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:40.261592 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.270565 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.315403 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.322970 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.331946 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.356406 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.399333 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.401037 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.404191 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.431103 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.431682 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.432991 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:40.437309 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:40.440662 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.450341 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.485661 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:40.498353 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.500422 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:40.507254 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:40.525859 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.540409 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.560017 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.563625 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:40.605661 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.639963 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:40.662803 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.670907 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:40.692534 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.711498 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.742846 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:40.747206 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:40.766505 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:40.786065 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.800824 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:40.830157 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.832206 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.890690 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.904630 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.921298 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:40.999915 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:41.029188 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:41.029510 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:41.034186 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:41.053133 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:41.073547 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:41.076167 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:41.090783 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:41.100729 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:10:41.139266 14403 raft_consensus.cc:482] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:41.139369 14403 raft_consensus.cc:504] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:41.139711 14403 leader_election.cc:288] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:41.141783 30213 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:41.143160 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:41.143358 30210 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:41.143447 30210 leader_election.cc:302] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:41.143693 14435 raft_consensus.cc:2726] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:41.144709 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:41.177588 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:41.186623 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:41.225176 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:41.242926 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:41.280112 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:41.298957 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:41.315557 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:41.346508 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:41.359550 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:41.380939 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:41.439100 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:41.449903 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:41.460001 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:41.462271 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:41.481114 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:41.594758 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:41.621284 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:41.633539 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:41.639750 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:41.703994 30344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 175 similar messages]
W20240501 14:10:41.704531 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:41.714143 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:41.719777 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:41.722445 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:41.773712 29948 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 154 similar messages]
W20240501 14:10:41.776732 30210 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 228 similar messages]
W20240501 14:10:41.777307 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:41.788645 32753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 182 similar messages]
W20240501 14:10:41.798149 30079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240501 14:10:41.827210 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:41.850543 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:41.854775 32359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:10:41.855060 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:41.855397 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:41.925501 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:42.002813 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:42.004500 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:42.009259 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:42.021728 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:42.022171 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:42.027495 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:42.068645 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:42.108273 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:42.111027 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:42.188334 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:42.220191 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:42.256017 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:42.290793 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:42.296090 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:42.335516 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:42.346487 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:42.347733 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:42.359726 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:42.408176 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:42.444379 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:42.475466 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:42.491292 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:10:42.511013 14433 raft_consensus.cc:482] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:42.511107 14433 raft_consensus.cc:504] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:42.511468 14433 leader_election.cc:288] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
W20240501 14:10:42.512274 29951 leader_election.cc:334] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:42.512405 29948 leader_election.cc:334] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:42.512485 29948 leader_election.cc:302] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:42.512602 14433 raft_consensus.cc:2726] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:42.528625 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:42.564910 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:42.569398 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:42.619251 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:42.621810 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:42.666219 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:42.678931 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:42.703809 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:42.725286 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:42.731698 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:42.779853 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:42.782553 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:42.806747 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:42.819417 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:42.839248 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:42.848446 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:42.886214 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:42.891371 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:42.896718 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:42.900367 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:42.907263 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:42.911548 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:42.921270 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:42.925806 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:42.977607 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:42.999034 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.013329 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:43.015239 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:43.033294 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.077710 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:43.119549 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:43.168263 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:43.171921 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.176563 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.253508 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.260632 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.281528 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.289834 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.308653 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:43.316033 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.331434 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.340325 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.370834 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.407894 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:43.426411 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:43.427345 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.463528 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:43.469871 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:43.478394 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.498811 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.506644 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.506809 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.525852 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:43.551862 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.572041 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.578325 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.636310 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.728857 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:43.752142 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.759588 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:43.764513 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:43.766039 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:43.769558 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.782058 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:43.864295 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:43.931617 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:43.958281 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:43.960940 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:10:43.974282 14449 raft_consensus.cc:482] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:43.974368 14449 raft_consensus.cc:504] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:43.974694 14449 leader_election.cc:288] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:43.975425 30213 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:43.975944 30210 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:43.976006 30210 leader_election.cc:302] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:43.976156 14449 raft_consensus.cc:2726] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:43.996192 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:44.010663 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:44.025696 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:44.025916 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:44.057577 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:44.101588 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:44.130383 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:44.223659 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:44.233301 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:44.279984 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:44.280978 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:44.283731 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:44.287236 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:44.293339 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:44.316294 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:44.346563 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:44.385843 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:44.418058 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:44.429102 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:44.432572 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:44.434581 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:44.453146 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:44.461586 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:44.477324 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:44.489209 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:44.527086 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:44.573194 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:44.622148 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:44.689586 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:44.708545 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:44.711004 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:44.716559 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:44.773308 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:44.836936 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:44.855551 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:44.858305 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:44.876546 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:44.906601 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:44.910840 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:44.932559 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:44.950327 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:44.974179 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:10:44.990401 14457 raft_consensus.cc:482] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:44.990494 14457 raft_consensus.cc:504] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:44.990866 14457 leader_election.cc:288] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:44.991726 30213 leader_election.cc:334] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:44.991896 30210 leader_election.cc:334] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:44.991971 30210 leader_election.cc:302] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:44.992084 14457 raft_consensus.cc:2726] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:45.021448 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:45.043502 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.077585 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.082444 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:45.119974 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.156160 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:45.203492 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.203827 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:45.216364 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.225713 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.233772 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.238204 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.249164 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.301059 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.307756 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:45.334944 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.349472 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.360010 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.388787 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:45.408725 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:45.442279 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:45.443003 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:45.491135 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.498617 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:45.520009 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:45.554653 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.559478 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.569087 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:45.569892 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.598948 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:45.600556 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:10:45.619828 14435 raft_consensus.cc:482] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:45.619938 14435 raft_consensus.cc:504] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:45.620406 14435 leader_election.cc:288] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
W20240501 14:10:45.621403 30210 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:45.621573 30213 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:45.621743 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025e07cbfd543e3b4722c6c69e4a6b3" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:10:45.621887 32415 raft_consensus.cc:2454] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
I20240501 14:10:45.622059 30213 leader_election.cc:302] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:45.622186 14435 raft_consensus.cc:2726] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:45.630019 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.678128 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.692656 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.702347 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.737475 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:45.743062 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:45.755523 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:45.765507 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.774905 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.809306 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.830159 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.834411 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.837025 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.851358 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.862510 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:10:45.875344 14455 raft_consensus.cc:482] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:45.875442 14455 raft_consensus.cc:504] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:10:45.875751 14455 leader_election.cc:288] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:45.876672 32753 leader_election.cc:334] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:45.876752 32750 leader_election.cc:334] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:45.876813 32750 leader_election.cc:302] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:45.876948 14455 raft_consensus.cc:2726] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:45.888316 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.906394 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:45.932904 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.935433 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.938586 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:45.950834 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:45.984472 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:45.999909 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:46.009399 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:46.224550 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:46.231693 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:46.237331 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:46.257938 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:46.265280 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:46.303004 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:46.319065 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:46.319279 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:46.363504 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:46.370380 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:46.372169 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:46.426098 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:46.471632 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:46.474694 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:46.507424 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:46.619256 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:46.658923 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:46.706148 30344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 179 similar messages]
W20240501 14:10:46.706179 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:46.706718 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:46.720412 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:46.733656 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:46.739748 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:46.780135 30213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 226 similar messages]
W20240501 14:10:46.780726 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:46.790827 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:46.805702 29948 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240501 14:10:46.815645 30079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240501 14:10:46.831501 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:46.831578 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:46.833168 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:46.849191 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:46.865509 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:46.869081 32356 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:10:46.880326 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:46.884568 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:46.899241 32753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 185 similar messages]
W20240501 14:10:46.905193 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:46.909664 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:46.934671 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:46.974828 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:47.002373 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:47.070031 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:47.082178 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:47.163055 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:47.172230 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:47.179499 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:47.180897 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:47.187906 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:47.232625 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:47.237455 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:47.242818 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:47.275759 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:47.295010 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:47.316851 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:47.373960 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:47.452093 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:47.459578 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:47.548292 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:47.572016 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:47.584757 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:47.600906 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:47.609184 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:47.655305 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:47.663367 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:47.670089 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:47.677583 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:47.681713 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:47.688292 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:47.694185 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:47.732426 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:47.742131 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:47.756431 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:47.769062 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:47.773152 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:47.808279 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:47.883751 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:47.884119 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:47.950708 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:47.956418 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:47.958693 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:47.969607 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:47.983907 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:47.996024 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:10:48.023200 14474 raft_consensus.cc:482] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:48.023315 14474 raft_consensus.cc:504] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:48.023674 14474 leader_election.cc:288] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
W20240501 14:10:48.024765 29948 leader_election.cc:334] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:48.025099 29951 leader_election.cc:334] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:48.025173 29951 leader_election.cc:302] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:48.025364 14474 raft_consensus.cc:2726] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:48.051856 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:48.062904 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:48.071514 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.091763 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.093017 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.117378 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.121634 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.139848 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.150465 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.153213 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:48.166692 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:48.194423 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:48.207718 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.214818 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:48.220934 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:48.245730 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.312310 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.388844 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.446986 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.475721 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.487600 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:48.506009 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:48.535339 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.556522 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:48.571975 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:48.590796 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.599679 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.619808 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.620396 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:48.637567 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.654523 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.664815 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.671801 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:48.724654 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.736148 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:48.746930 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.762571 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:48.799620 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:10:48.806424 14439 raft_consensus.cc:482] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:48.806524 14439 raft_consensus.cc:504] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:48.807091 14439 leader_election.cc:288] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:10:48.807477 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce3093e24347f3a4e121af452c6361" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:10:48.807607 30407 raft_consensus.cc:2454] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
W20240501 14:10:48.808286 32359 leader_election.cc:341] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Not found: Tablet not found: 89ce3093e24347f3a4e121af452c6361
W20240501 14:10:48.809011 32359 leader_election.cc:334] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:48.809262 32359 leader_election.cc:302] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3
I20240501 14:10:48.809425 14419 raft_consensus.cc:2726] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:48.816670 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:48.840961 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:48.867560 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:48.873606 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.875705 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:48.912910 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.964939 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:48.965575 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:49.030344 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:49.038288 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:49.089368 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:49.094800 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:49.140276 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:10:49.167405 14455 raft_consensus.cc:482] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:49.167542 14455 raft_consensus.cc:504] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:49.167935 14455 leader_election.cc:288] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:49.168874 32750 leader_election.cc:334] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:49.168957 32753 leader_election.cc:334] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:49.169044 32753 leader_election.cc:302] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:49.169160 14455 raft_consensus.cc:2726] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:49.190173 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:49.214231 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:49.214704 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:49.251370 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:49.306829 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:49.339957 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:49.360461 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:49.369246 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:49.384410 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:49.417980 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:49.453058 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:49.476651 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:49.477633 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:49.486830 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:49.507673 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:49.536681 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:49.542877 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:49.589625 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:49.595829 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:49.618590 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:49.625945 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:49.641045 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:49.656091 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:49.670913 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:49.689265 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:49.718178 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:49.752226 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:49.781739 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:49.810768 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:49.842283 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:49.868146 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:49.896981 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:49.941023 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:49.966934 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.077642 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:50.099449 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:50.109782 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.118235 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.128178 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:50.130193 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:50.165243 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:50.168025 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.184605 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:50.241415 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:50.242704 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.266494 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.272373 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:50.285950 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.337057 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:50.347537 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.349388 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:50.350271 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.362536 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:50.366755 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.437263 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:50.442755 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:50.452284 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:50.467046 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.493593 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.508131 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.544387 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.577332 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.589767 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:50.638336 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.674237 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.683950 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:50.716864 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.723765 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.733609 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:50.770077 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:50.775383 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.836284 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.845127 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:50.866284 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.874410 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.892513 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:50.893514 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:50.920249 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:51.037665 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:51.048956 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:10:51.089378 14477 raft_consensus.cc:482] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:10:51.089660 14477 raft_consensus.cc:504] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:10:51.090214 14477 leader_election.cc:288] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:10:51.090405 32415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce3093e24347f3a4e121af452c6361" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:10:51.090555 32415 raft_consensus.cc:2454] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
W20240501 14:10:51.091130 30344 leader_election.cc:341] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Not found: Tablet not found: 89ce3093e24347f3a4e121af452c6361
W20240501 14:10:51.091223 30344 leader_election.cc:334] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:51.091256 30344 leader_election.cc:302] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3
I20240501 14:10:51.091348 14477 raft_consensus.cc:2726] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:51.093076 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:51.110891 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:51.143399 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:51.153168 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:51.166720 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:51.181499 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:51.188565 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:51.191001 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:51.195976 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:51.200984 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:51.226553 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:51.263607 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:51.312973 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:51.319087 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:51.319561 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:51.319922 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:51.321573 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:51.323691 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:51.350819 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:51.381035 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:51.389034 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:51.398449 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:51.420614 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:51.472396 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:51.503479 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:51.536754 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:51.542255 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:51.596272 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:51.604115 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:51.698575 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:51.738615 30344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 183 similar messages]
W20240501 14:10:51.744411 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:51.750806 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:51.756551 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:51.760370 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:51.788537 30210 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 217 similar messages]
W20240501 14:10:51.821681 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:51.822503 29951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240501 14:10:51.833864 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:51.855141 30082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:10:51.869077 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:51.871974 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:51.884500 32359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240501 14:10:51.892688 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:51.926499 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:51.940119 32750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 181 similar messages]
W20240501 14:10:51.967841 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:52.013490 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:52.039454 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:52.073887 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:52.091184 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:52.097023 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:52.124519 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:52.126060 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:52.139456 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:52.145428 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:52.160683 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:52.197199 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:52.198679 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:52.212312 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:52.215566 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:52.221978 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:52.230571 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:52.233361 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:52.317719 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:52.324043 14448 raft_consensus.cc:482] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:52.324151 14448 raft_consensus.cc:504] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:52.324566 14448 leader_election.cc:288] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:52.325556 29951 leader_election.cc:334] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:52.325680 29948 leader_election.cc:334] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:52.325752 29948 leader_election.cc:302] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:52.325875 14448 raft_consensus.cc:2726] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:52.342756 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:52.367802 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:52.381047 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:52.398984 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:52.485597 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:52.506420 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:52.539876 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:52.562013 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:52.577406 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:52.622329 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:52.646541 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:52.654742 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:52.669159 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:52.669910 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:52.713269 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:52.754297 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:52.771852 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:52.799674 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:52.824165 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:52.824716 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:52.872790 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:52.899936 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:52.923475 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:52.931733 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:52.937952 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:52.947901 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:52.990330 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.000054 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:53.050001 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.051224 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:53.073091 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.075520 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:53.078081 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.132653 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.135286 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.164356 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.193584 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:53.201298 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.248690 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:53.258406 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.274570 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.332260 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.356002 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.359038 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:53.479701 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.492153 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:53.497406 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.530880 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:53.555922 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:53.577095 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:53.605676 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.621623 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.665690 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:53.666994 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.685021 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.694392 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:53.724028 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.724762 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:53.730365 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.740106 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.751174 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:53.759928 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:53.782308 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.787067 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:53.826225 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:53.829958 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:53.831000 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:53.846884 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:53.874783 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.885658 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:53.905632 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:53.911912 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:53.957384 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:54.018270 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:54.036583 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:54.037341 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:54.045507 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:54.046669 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:54.089833 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:54.106194 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:54.138092 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:54.150585 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:54.191694 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:54.204581 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:54.259797 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:54.291180 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:54.300181 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:54.315366 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:54.327234 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:54.367920 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:54.444777 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:54.463980 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:54.499184 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:54.505263 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:54.524359 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:54.528190 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:54.550179 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:54.562106 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:54.572687 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:54.604956 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:54.683580 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:54.709156 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:54.734529 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:54.751935 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:54.778074 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:54.794377 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:54.800431 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:54.826071 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:54.826347 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:54.834401 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:54.851426 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:54.861316 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:54.902910 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:54.906250 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:54.928920 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:54.938347 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:54.986862 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:54.994709 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:55.034005 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:55.071943 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:55.085237 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:55.085659 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:55.146404 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:55.158205 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:10:55.220317 14505 raft_consensus.cc:482] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:55.220430 14505 raft_consensus.cc:504] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:55.220767 14505 leader_election.cc:288] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:55.221580 30213 leader_election.cc:334] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:55.221712 30210 leader_election.cc:334] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:55.221789 30210 leader_election.cc:302] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:55.221918 14505 raft_consensus.cc:2726] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:55.245472 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:55.286166 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:55.295629 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:55.349510 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:55.379701 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:55.382938 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:55.391993 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:55.454648 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:55.466219 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:55.483249 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:55.512569 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:55.513526 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:55.525166 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:55.557582 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:55.558674 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:55.571569 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:55.621170 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:55.626324 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:55.672451 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:55.675225 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:55.723407 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:55.724877 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:55.731580 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:55.784046 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:55.795280 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:55.807808 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:55.809320 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:55.851101 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:55.851768 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:55.870738 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:55.870738 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:55.877033 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:55.924072 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:55.987354 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:56.023696 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:56.024381 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:56.026880 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:56.034426 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:56.085135 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:56.087155 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:56.106515 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:56.113638 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:56.175678 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:56.204644 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:56.218178 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:56.225435 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:56.238596 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:56.250255 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:56.257120 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:10:56.260731 14509 raft_consensus.cc:482] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:56.260838 14509 raft_consensus.cc:504] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:10:56.261173 14509 leader_election.cc:288] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
W20240501 14:10:56.262073 29951 leader_election.cc:334] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:56.262431 29948 leader_election.cc:334] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:56.262507 29948 leader_election.cc:302] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:56.262753 14509 raft_consensus.cc:2726] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:56.292301 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:10:56.319658 14504 raft_consensus.cc:482] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:56.319764 14504 raft_consensus.cc:504] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:56.320273 14504 leader_election.cc:288] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:10:56.320576 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025e07cbfd543e3b4722c6c69e4a6b3" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:10:56.320721 30277 raft_consensus.cc:2454] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
W20240501 14:10:56.321210 32356 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:56.321328 32359 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:56.321405 32359 leader_election.cc:302] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:56.321777 14504 raft_consensus.cc:2726] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:56.356912 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:56.382072 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:56.389858 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:56.400987 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:56.423319 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:56.427364 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:56.435671 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:56.486268 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:56.494374 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:56.494927 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:56.508852 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:56.516326 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:56.527568 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:56.546550 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:56.573962 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:56.580353 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:10:56.581454 14486 raft_consensus.cc:482] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:56.581540 14486 raft_consensus.cc:504] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:10:56.581872 14486 leader_election.cc:288] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:56.582592 32753 leader_election.cc:334] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:56.582674 32750 leader_election.cc:334] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:56.582727 32750 leader_election.cc:302] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:56.582859 14486 raft_consensus.cc:2726] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:56.671072 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:56.691954 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:56.701200 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:56.702910 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:56.731148 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:56.737972 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:56.776469 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:56.790207 30213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 221 similar messages]
W20240501 14:10:56.839674 29948 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 158 similar messages]
W20240501 14:10:56.847076 30341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 185 similar messages]
W20240501 14:10:56.867465 30079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240501 14:10:56.871364 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:56.889907 32356 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240501 14:10:56.894178 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:56.915434 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:56.925220 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:56.935766 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:10:56.986959 14517 raft_consensus.cc:482] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:56.987074 14517 raft_consensus.cc:504] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:10:56.987421 14517 leader_election.cc:288] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:56.988428 30213 leader_election.cc:334] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:56.988521 30210 leader_election.cc:334] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:10:56.988605 30210 leader_election.cc:302] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:56.988729 14517 raft_consensus.cc:2726] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:10:56.993414 32750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 183 similar messages]
W20240501 14:10:57.002964 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:57.068625 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:57.094692 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:57.115079 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:57.132350 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:57.147984 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:57.163044 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:57.190202 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:57.198797 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:57.198849 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:57.201169 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:57.228473 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:57.245461 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:57.260025 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:57.289166 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:57.336256 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:57.343602 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:57.354449 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:57.360423 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:57.381491 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:57.390113 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:57.443744 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:57.466181 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:57.552733 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:57.569882 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:10:57.591121 14520 raft_consensus.cc:482] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:10:57.591256 14520 raft_consensus.cc:504] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:10:57.591735 14520 leader_election.cc:288] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:10:57.592798 30210 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:57.593097 30213 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:57.593215 30213 leader_election.cc:302] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:10:57.593369 14520 raft_consensus.cc:2726] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:57.593845 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:57.598377 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:57.603595 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:57.644737 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:57.664541 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:57.685860 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:57.690402 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:57.774117 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:57.804333 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:57.828198 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:57.849960 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:57.850318 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:57.865692 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:57.919880 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:57.933193 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:57.953420 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:57.972607 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:57.983541 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:57.994768 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:57.996574 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.036278 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.055737 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.066547 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:58.073591 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.084872 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:58.120977 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.187103 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.231007 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.253845 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.287158 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:58.301024 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.307500 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:58.339440 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.395110 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.437078 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.461974 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:58.475548 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.484802 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.502523 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:58.530061 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.546319 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:58.549186 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:58.568724 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.573259 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.591646 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:58.593104 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:58.608415 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:58.614084 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.648967 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:58.659289 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:58.707999 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.759974 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.764178 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:58.767702 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.825363 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.838397 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.854902 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:58.878016 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:58.880862 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:58.890436 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.892398 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:58.901683 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.901751 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:58.916383 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:58.972869 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:58.983043 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:58.984010 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:58.984087 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:59.037025 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:59.051501 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:59.054061 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:59.064570 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:59.068424 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:59.077544 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:59.083856 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:59.130390 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:59.167842 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:59.196452 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:59.224735 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:59.279094 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:59.330943 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:59.339474 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:59.364591 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:59.417690 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:59.431936 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:59.518211 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:59.535952 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:59.547570 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:59.557399 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:59.573328 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:59.585013 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:59.594857 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:59.644515 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:59.677029 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:59.692903 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:59.710111 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:59.719563 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:59.764943 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:59.798748 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:10:59.844638 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:59.892988 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:59.920842 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:59.928541 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:10:59.931336 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:59.944574 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:59.972908 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:59.976332 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:59.989600 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:00.012667 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:00.052430 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:00.130018 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:00.168969 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:00.203343 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:00.242367 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:00.273378 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:00.311538 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:00.321682 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:00.328488 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:00.333632 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:00.337055 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:00.337224 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:00.355345 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:00.394831 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:00.396090 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:00.421339 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:00.453418 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:00.462070 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:00.478072 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:00.487555 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:00.512555 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:00.516644 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:00.550086 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:00.561316 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:00.565320 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:00.597136 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:00.614403 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:00.653780 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:00.718762 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:00.755306 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:00.800488 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:00.828326 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:00.881489 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:00.882990 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:00.890517 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:00.900871 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:00.906404 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:00.949935 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:00.975608 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:00.992158 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:01.009109 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:01.017143 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:01.085903 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:01.086714 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:01.103253 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:01.165678 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:01.166620 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:01.204463 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:01.219395 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:01.249181 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:01.255277 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:01.302194 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:01.335570 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:01.339254 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:01.355871 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:01.360764 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:01.362447 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:01.380268 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:01.390452 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:01.473889 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:01.491132 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:01.514364 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:01.519650 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:01.521560 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:01.536849 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:01.578222 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:01.579147 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:01.580114 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:01.581236 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:01.589453 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:01.619050 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:01.621467 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:01.631984 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:01.640623 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:01.660133 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:01.660220 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:01.667973 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:01.730186 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:11:01.733779 14520 raft_consensus.cc:482] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:11:01.733920 14520 raft_consensus.cc:504] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:11:01.734333 14520 leader_election.cc:288] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265)
I20240501 14:11:01.734717 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025e07cbfd543e3b4722c6c69e4a6b3" candidate_uuid: "1d0bb69611af4876a2b3b33c4766e13a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:11:01.734864 32416 raft_consensus.cc:2454] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0bb69611af4876a2b3b33c4766e13a in term 1.
W20240501 14:11:01.735369 30210 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:01.735502 30213 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:11:01.735554 30210 leader_election.cc:302] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:11:01.735679 14520 raft_consensus.cc:2726] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:11:01.753894 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:01.775274 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:01.816191 30210 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 230 similar messages]
W20240501 14:11:01.832631 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:01.843729 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:01.853451 30344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 183 similar messages]
W20240501 14:11:01.888289 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:01.896950 30079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:11:01.913995 29951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 147 similar messages]
W20240501 14:11:01.926597 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:01.933959 32359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240501 14:11:01.952008 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:01.952976 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:02.012418 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:02.014755 32750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 179 similar messages]
W20240501 14:11:02.016688 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:02.038429 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:02.076155 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:02.089622 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:02.102401 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:02.116989 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:02.120486 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:02.136780 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:02.159595 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:02.177311 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:02.244472 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:02.255066 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:02.286125 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:02.349676 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:02.364262 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:02.404153 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:02.412958 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:02.480911 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:02.506955 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:02.515386 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:02.544440 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:02.560950 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:02.563683 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:02.586444 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:02.626791 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:02.635718 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:02.669425 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:02.677351 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:02.677577 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:02.735972 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:02.741240 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:02.767416 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:02.789794 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:11:02.825138 14520 raft_consensus.cc:482] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:11:02.825258 14520 raft_consensus.cc:504] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:11:02.825588 14520 leader_election.cc:288] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:11:02.826503 30213 leader_election.cc:334] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:02.826602 30210 leader_election.cc:334] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:11:02.826653 30210 leader_election.cc:302] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:11:02.826781 14520 raft_consensus.cc:2726] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:11:02.828537 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:02.831710 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:02.861536 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:02.862829 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:02.885306 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:02.918737 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:02.941658 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:02.974870 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:02.994480 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:02.997093 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:03.011471 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:03.018355 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:03.028471 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:03.049744 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:03.075861 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:03.145402 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:03.196938 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:03.203228 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:03.211827 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:03.229156 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:03.275995 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:03.297245 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:03.325623 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:03.425311 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:03.445536 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:03.476826 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:03.481886 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:03.491739 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:03.495975 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:03.504099 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:03.531234 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:03.572069 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:03.577008 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:03.617826 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:03.618856 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:03.624650 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:03.646611 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:03.656353 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:03.664645 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:03.670701 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:03.718945 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:03.724599 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:03.758004 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:03.787376 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:03.810405 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:03.817080 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:03.822628 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:03.891098 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:03.954203 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:03.956305 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:11:03.989923 14537 raft_consensus.cc:482] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:11:03.990041 14537 raft_consensus.cc:504] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } }
I20240501 14:11:03.990379 14537 leader_election.cc:288] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:11:03.991521 32753 leader_election.cc:334] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:03.991801 32750 leader_election.cc:334] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:11:03.991868 32750 leader_election.cc:302] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:11:03.992576 14537 raft_consensus.cc:2726] T 6229c0c3e98c462b95f9bfe066367e87 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:11:04.013919 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:04.032034 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:04.032495 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:04.038110 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:04.042932 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:04.091099 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:04.096055 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:04.097592 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:04.109609 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:04.114416 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:04.116430 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:04.178478 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:04.200675 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:04.217557 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:04.223737 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:04.256031 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:04.272738 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:04.305845 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:04.314666 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:04.324283 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:04.355659 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:11:04.358000 14552 raft_consensus.cc:482] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:11:04.358134 14552 raft_consensus.cc:504] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:11:04.358597 14552 leader_election.cc:288] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a91029b90bf9415a89a2f70736759ad3 (127.22.252.133:46265), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:11:04.358855 32416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce3093e24347f3a4e121af452c6361" candidate_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a91029b90bf9415a89a2f70736759ad3" is_pre_election: true
I20240501 14:11:04.359001 32416 raft_consensus.cc:2454] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c217469ae4478ebd49e4ed9f7352d7 in term 1.
W20240501 14:11:04.359529 30344 leader_election.cc:334] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:04.359683 30344 leader_election.cc:341] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Not found: Tablet not found: 89ce3093e24347f3a4e121af452c6361
I20240501 14:11:04.359761 30344 leader_election.cc:302] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3
I20240501 14:11:04.359915 14552 raft_consensus.cc:2726] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:11:04.361439 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:04.365830 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:04.395855 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:04.461228 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:04.525677 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:04.566409 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:04.569958 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:04.574393 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:04.618795 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:04.662621 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:04.673848 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:04.687395 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:04.706578 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:11:04.736001 14544 raft_consensus.cc:482] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:11:04.736129 14544 raft_consensus.cc:504] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:11:04.736517 14544 leader_election.cc:288] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:11:04.737470 29951 leader_election.cc:334] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:04.737560 29948 leader_election.cc:334] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:11:04.737651 29948 leader_election.cc:302] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:11:04.737797 14544 raft_consensus.cc:2726] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:11:04.748517 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:04.749388 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:04.750699 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:04.795487 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:04.816494 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:04.834417 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:04.871451 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:04.876595 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:04.895954 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:04.929468 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:04.937716 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:04.942255 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:04.945770 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:05.001345 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:05.009043 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:05.031726 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:05.044515 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:05.046684 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:05.052546 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:05.053283 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:05.136274 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:05.179291 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:05.192914 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:05.198156 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:05.242275 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:05.251132 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:05.259217 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:05.353837 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:05.356995 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:05.370594 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:05.407497 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:05.408286 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:05.444669 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:05.449169 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:05.489539 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:05.495187 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:05.496922 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:05.518090 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:05.549767 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:05.571660 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:05.580516 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:05.657521 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:05.673111 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:05.701992 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:05.759442 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:05.796144 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:05.853209 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:05.891424 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:05.897066 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:05.909651 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:05.913432 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:05.917793 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:05.950757 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:05.970095 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:05.976595 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:05.998003 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:06.010026 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:06.030944 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:06.036375 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:06.051369 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:06.059619 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:06.079285 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:06.110625 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:06.119946 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:06.150391 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:06.181118 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:06.208619 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:06.240639 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:06.252498 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:06.321667 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:06.347676 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:06.355221 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:06.357246 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:06.369756 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:06.397210 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:06.423463 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:11:06.428462 14551 raft_consensus.cc:482] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:11:06.428618 14551 raft_consensus.cc:504] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:11:06.429190 14551 leader_election.cc:288] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:11:06.429399 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce3093e24347f3a4e121af452c6361" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:11:06.429512 30406 raft_consensus.cc:2454] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
W20240501 14:11:06.430521 32359 leader_election.cc:341] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Not found: Tablet not found: 89ce3093e24347f3a4e121af452c6361
W20240501 14:11:06.431922 32359 leader_election.cc:334] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:11:06.432134 32359 leader_election.cc:302] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3
I20240501 14:11:06.432543 14551 raft_consensus.cc:2726] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:11:06.474265 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:06.484177 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:06.484509 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:06.525149 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:06.547870 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:06.561810 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:06.576311 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:06.616132 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:06.626359 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:06.647846 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:06.650543 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:06.679314 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:06.684721 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:06.693615 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:06.702033 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:06.715677 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:06.751603 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:06.792593 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:06.795356 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:06.821938 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:11:06.834645 14569 raft_consensus.cc:482] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:11:06.834756 14569 raft_consensus.cc:504] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:11:06.835134 14569 leader_election.cc:288] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:11:06.836055 32753 leader_election.cc:334] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:06.836144 32750 leader_election.cc:334] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:11:06.836254 32750 leader_election.cc:302] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:11:06.836395 14569 raft_consensus.cc:2726] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:11:06.851429 30210 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 220 similar messages]
W20240501 14:11:06.871589 30341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 178 similar messages]
W20240501 14:11:06.880649 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:06.917496 29948 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 157 similar messages]
W20240501 14:11:06.918871 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:06.936846 32356 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240501 14:11:06.941394 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:07.005034 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:07.055119 30079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240501 14:11:07.055963 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:07.059768 32753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 181 similar messages]
W20240501 14:11:07.085919 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:07.086824 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:07.091712 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:07.134469 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:07.148260 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:07.164227 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:07.194183 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:07.206310 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:07.213366 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:07.219574 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:07.227758 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:07.240613 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:07.278307 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:07.319453 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:07.329975 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:07.331671 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:07.393281 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:07.394173 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:07.450258 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:07.514324 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:07.527643 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:07.531888 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:07.536912 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:07.543143 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:07.548015 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:07.585779 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:07.602645 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:07.624814 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:07.625428 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:07.633307 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:07.644038 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:07.725998 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:07.726284 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:07.732029 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:07.780161 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:07.781508 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:07.781452 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:07.800839 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:07.836071 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:07.836758 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:07.880750 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:07.885162 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:07.943249 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:07.948554 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:08.014079 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:08.015507 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:08.044124 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:08.152338 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:08.186821 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:08.194340 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:08.247440 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:08.251457 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:08.260388 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:08.286011 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:08.290279 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:08.308530 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:08.320937 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:08.337203 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:08.340374 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:08.343546 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:08.385123 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:08.389189 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:08.421242 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:08.451337 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:08.487192 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:08.526307 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:08.535861 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:08.558724 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:08.579730 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:08.729923 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:08.732534 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:08.738422 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:08.757315 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:08.774071 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:08.783988 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:08.808312 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:08.825593 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:08.826537 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:08.846062 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:08.889366 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:08.927956 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:08.941921 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:08.972399 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:08.979606 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:09.008666 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:09.011189 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:09.011420 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:09.028589 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:09.068053 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:09.070945 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:09.072270 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:09.104692 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:09.105528 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:09.184409 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:09.200318 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:09.224939 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:09.231704 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:09.242759 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:09.271423 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:09.284014 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:09.302613 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:09.303931 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:09.324004 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:09.328224 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:09.413684 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:09.418870 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:09.451773 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:09.490550 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:09.520848 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:09.551585 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:09.558050 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:09.599668 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:09.604506 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:09.639361 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:09.656244 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:09.668061 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:09.698635 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:09.713685 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:09.740608 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:09.758792 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:09.795307 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:09.803491 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:09.832564 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:09.847730 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:09.862655 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:09.870680 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:09.871062 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:09.873965 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:09.910635 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:09.962697 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:09.969034 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:09.990680 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:10.014559 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:10.082930 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:10.126447 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:10.162078 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:10.197774 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:10.223595 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:10.231781 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:10.233829 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:10.236312 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:10.245762 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:10.251084 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:10.251543 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:10.274852 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:10.294549 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:10.297775 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:10.339308 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:10.340191 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:10.349304 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:10.367601 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:10.379874 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:10.396701 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:10.397313 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:10.425328 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:11:10.474804 14520 raft_consensus.cc:482] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:11:10.474910 14520 raft_consensus.cc:504] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:11:10.475272 14520 leader_election.cc:288] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:11:10.476179 30213 leader_election.cc:334] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:10.476497 30210 leader_election.cc:334] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:11:10.476603 30210 leader_election.cc:302] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:11:10.476744 14520 raft_consensus.cc:2726] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:11:10.516865 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:10.579404 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:10.612535 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:10.618146 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:10.661229 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:10.683344 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:10.691268 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:10.697014 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:10.722971 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:10.739735 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:10.773125 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:10.807232 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:10.819182 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:10.846185 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:10.911029 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:10.911504 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:10.949653 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:10.978927 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:11.034435 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:11.046536 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:11.049738 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:11.063689 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:11.069223 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:11.186360 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:11.187245 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:11.209510 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:11.211097 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:11.214380 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:11.246161 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:11.250799 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:11.277172 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:11.286993 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:11.345237 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:11.345312 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:11.367889 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:11.373121 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:11.386337 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:11.448676 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:11.454288 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:11.521242 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:11.530601 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:11.586609 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:11.591001 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:11.602556 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:11.618284 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:11.621165 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:11.625453 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:11.683370 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:11.732044 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:11.753357 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:11.770118 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:11.782609 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:11.813797 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:11.830171 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:11.850762 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:11.854466 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:11.861397 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:11.862179 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:11.868108 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:11.871698 30213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 224 similar messages]
W20240501 14:11:11.898789 30344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 182 similar messages]
W20240501 14:11:11.930508 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:11.956669 32359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:11:11.981088 29951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240501 14:11:11.993605 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:12.035540 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:12.040735 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:12.059932 32750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 179 similar messages]
W20240501 14:11:12.060609 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:12.062911 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:12.101614 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:12.122159 30079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240501 14:11:12.129276 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:12.141728 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:12.146268 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:12.153676 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:12.171417 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:12.190516 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:12.206293 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:12.222478 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:11:12.225461 14595 raft_consensus.cc:482] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:11:12.225559 14595 raft_consensus.cc:504] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:11:12.225893 14595 leader_election.cc:288] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:11:12.226821 30213 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:12.227006 30210 leader_election.cc:334] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:11:12.227072 30210 leader_election.cc:302] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:11:12.227233 14595 raft_consensus.cc:2726] T 21efaac3a3d643679c5abaaddb9e5a3a P 1d0bb69611af4876a2b3b33c4766e13a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:12.237587 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:12.277051 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:12.278223 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:12.314181 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:12.341970 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:12.374234 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:12.379869 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:12.392843 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:11:12.414762 14608 raft_consensus.cc:482] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:11:12.414872 14608 raft_consensus.cc:504] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:11:12.415221 14608 leader_election.cc:288] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497)
W20240501 14:11:12.416072 29948 leader_election.cc:334] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:12.416157 29951 leader_election.cc:334] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:11:12.416249 29951 leader_election.cc:302] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:11:12.416378 14608 raft_consensus.cc:2726] T 0cccc8eec30141c1946b57d3be27000f P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:11:12.448889 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:12.457631 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:12.484091 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:12.511886 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:12.518357 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:12.539791 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:12.559036 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:12.603708 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:12.608633 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:12.626814 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:12.644694 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:12.646255 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:12.657487 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:12.658491 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:12.708257 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:12.709761 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:12.728933 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:12.732564 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:12.738992 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:12.782579 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:12.792459 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:12.801854 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:12.825717 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:12.835906 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:12.838266 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:12.849514 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:12.867718 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:12.870296 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:12.872503 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:12.956048 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:12.991173 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:13.114041 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:13.153221 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:13.177145 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:13.204789 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:13.223851 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:13.265810 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:13.281153 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:13.317693 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:13.331970 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:13.366632 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:13.372977 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:13.379105 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:13.458226 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:13.487514 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:13.491498 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:13.502725 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:13.547886 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:13.554900 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:13.592689 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:13.639747 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:13.648620 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:13.657809 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:13.688292 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:13.707621 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:11:13.730827 14610 raft_consensus.cc:482] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:11:13.730939 14610 raft_consensus.cc:504] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ecd907cf5634ac98f3f54f3dbec6849" member_type: VOTER last_known_addr { host: "127.22.252.136" port: 45937 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:11:13.731280 14610 leader_election.cc:288] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:11:13.732311 32750 leader_election.cc:334] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:13.732616 32753 leader_election.cc:334] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:11:13.732672 32753 leader_election.cc:302] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ecd907cf5634ac98f3f54f3dbec6849; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:11:13.732821 14610 raft_consensus.cc:2726] T 1ca6bdb2e17a4e5e9dab845cad66ada8 P 9ecd907cf5634ac98f3f54f3dbec6849 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:11:13.748359 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:13.794903 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:13.817606 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:13.850586 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:13.854766 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:13.856775 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:13.858036 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:13.873211 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:13.917539 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:13.924163 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:13.950999 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:14.000803 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:14.016780 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:14.056380 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:14.075312 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:14.083458 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:14.085168 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:14.138669 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:14.140573 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:14.179725 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:14.219486 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:14.235260 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:14.244766 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:14.272195 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:14.324190 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:14.346153 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:14.349851 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:14.351528 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:14.386932 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:14.387010 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:14.388564 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:14.420550 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:14.499528 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:14.535735 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:14.572060 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:14.595518 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:14.598183 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:14.604008 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:14.617400 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:14.637317 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:14.671083 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:14.684331 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:14.694039 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:14.698127 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:14.805644 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:14.827065 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:14.855008 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:14.856570 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:14.866153 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:14.871555 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:14.871735 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:14.874570 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:14.884421 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:14.892781 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:14.911175 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:14.950345 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:14.964550 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:14.974500 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:14.992055 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:15.070865 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:15.076110 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:15.093786 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:15.109185 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:15.148618 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:15.158329 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:15.165216 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:15.166801 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:15.179560 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:15.199991 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:15.208405 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:15.213972 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:15.223824 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:15.236801 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:15.254515 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:15.275115 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:15.280205 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:15.283752 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:15.286778 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:15.320420 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:11:15.333225 29862 maintenance_manager.cc:419] P f2cd510126964a349f8d7712a2b33cf5: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033371
I20240501 14:11:15.366103 29861 maintenance_manager.cc:643] P f2cd510126964a349f8d7712a2b33cf5: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.032s user 0.022s sys 0.009s Metrics: {"bytes_written":112430,"cfile_init":1,"dirs.queue_time_us":175,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":900,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1011,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":309,"thread_start_us":151,"threads_started":2,"wal-append.queue_time_us":140}
I20240501 14:11:15.366976 29862 maintenance_manager.cc:419] P f2cd510126964a349f8d7712a2b33cf5: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 38562 bytes on disk
I20240501 14:11:15.367465 29861 maintenance_manager.cc:643] P f2cd510126964a349f8d7712a2b33cf5: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
W20240501 14:11:15.377761 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:15.398432 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:15.402029 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:15.531113 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:15.565677 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:15.584133 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:15.606460 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:11:15.620821 14627 raft_consensus.cc:482] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:11:15.621038 14627 raft_consensus.cc:504] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:11:15.621750 14627 leader_election.cc:288] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:11:15.625334 29948 leader_election.cc:334] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:15.625505 29951 leader_election.cc:334] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:11:15.625661 29951 leader_election.cc:302] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:11:15.628156 14627 raft_consensus.cc:2726] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:11:15.633453 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:15.638718 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:15.671695 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:15.695977 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:15.723096 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:15.801028 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:15.810675 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:15.882014 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:15.884992 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:15.885991 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:15.889170 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:15.907302 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:15.925987 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:15.939105 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:16.007202 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:16.059230 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:16.086002 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:16.088145 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:16.125509 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:11:16.130987 14633 raft_consensus.cc:482] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:11:16.131098 14633 raft_consensus.cc:504] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } }
I20240501 14:11:16.132055 14633 leader_election.cc:288] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), 1d0bb69611af4876a2b3b33c4766e13a (127.22.252.131:43103)
I20240501 14:11:16.132452 30277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025e07cbfd543e3b4722c6c69e4a6b3" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d0bb69611af4876a2b3b33c4766e13a" is_pre_election: true
I20240501 14:11:16.132587 30277 raft_consensus.cc:2454] T 5025e07cbfd543e3b4722c6c69e4a6b3 P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
W20240501 14:11:16.132773 32356 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:16.133368 32359 leader_election.cc:334] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:11:16.133440 32359 leader_election.cc:302] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a, a91029b90bf9415a89a2f70736759ad3; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:11:16.133562 14633 raft_consensus.cc:2726] T 5025e07cbfd543e3b4722c6c69e4a6b3 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:11:16.168658 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:16.183944 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:16.190510 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:16.211846 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:16.227772 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:16.263216 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:16.289454 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:16.307257 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:16.415485 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:16.418629 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:16.420821 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:16.429818 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:16.431722 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:16.488775 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:16.508303 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:16.518631 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:16.518862 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:16.543942 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:16.585505 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:16.619402 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:16.630419 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:16.656265 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:16.668203 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:16.686425 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:16.688894 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:16.693497 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:16.699152 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:16.764923 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:16.782215 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:16.831002 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:16.849211 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:16.873260 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:16.910789 30213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 221 similar messages]
W20240501 14:11:16.922317 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:16.940882 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:16.967118 30341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 179 similar messages]
W20240501 14:11:16.968550 32359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:11:16.977100 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:16.983052 29948 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240501 14:11:17.007783 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:17.021417 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:17.024505 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:17.036598 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:17.058610 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:17.107838 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:17.128701 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:17.130446 30082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240501 14:11:17.131131 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:17.134120 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:17.147199 32753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111) [suppressed 185 similar messages]
W20240501 14:11:17.160526 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:17.220991 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:17.288339 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:17.312098 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:17.338744 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:17.345743 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:17.357121 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:17.377122 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:17.381460 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:17.383476 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:17.401695 32750 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:17.428011 29951 consensus_peers.cc:485] T eb828a6a5cd240f5acb591e7eae5e1a8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:17.441295 30213 consensus_peers.cc:485] T c5d406c64b2e432a88e8c675b91dd144 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:17.448102 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:17.451388 30341 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:17.457716 30079 consensus_peers.cc:485] T 3125c67fe40641c2bfd5dcac81e7237f P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:17.469974 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:17.476570 32359 consensus_peers.cc:485] T 776d6722a14345f490f69f90d80d6a6a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:17.513443 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:17.515040 30344 consensus_peers.cc:485] T 8bb9687045d44efa888a69c7acde1105 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:11:17.535378 14627 raft_consensus.cc:482] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:11:17.535491 14627 raft_consensus.cc:504] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } } peers { permanent_uuid: "93f2518fbadf4f9991cda93afacb363e" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 41387 } }
I20240501 14:11:17.535905 14627 leader_election.cc:288] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:11:17.536438 30079 consensus_peers.cc:485] T 529f938305214c3ba33f61e06d1acd77 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:17.536808 29951 leader_election.cc:334] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:17.536924 29948 leader_election.cc:334] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:11:17.536994 29948 leader_election.cc:302] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93f2518fbadf4f9991cda93afacb363e; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:11:17.537128 14627 raft_consensus.cc:2726] T 65bc0e6ad1fa491e8e5ae225f86561f9 P 93f2518fbadf4f9991cda93afacb363e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:11:17.555437 14634 raft_consensus.cc:482] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db935501e9104de5b15c8780681d6c38)
I20240501 14:11:17.555565 14634 raft_consensus.cc:504] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bb69611af4876a2b3b33c4766e13a" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 43103 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "db935501e9104de5b15c8780681d6c38" member_type: VOTER last_known_addr { host: "127.22.252.135" port: 41915 } }
I20240501 14:11:17.555948 14634 leader_election.cc:288] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915)
W20240501 14:11:17.556885 30213 leader_election.cc:334] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:17.556979 30210 leader_election.cc:334] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
I20240501 14:11:17.557057 30210 leader_election.cc:302] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0bb69611af4876a2b3b33c4766e13a; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, db935501e9104de5b15c8780681d6c38
I20240501 14:11:17.557205 14634 raft_consensus.cc:2726] T 36bdef51cf8a4256bedd1102d04c7aef P 1d0bb69611af4876a2b3b33c4766e13a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:11:17.560415 32356 consensus_peers.cc:485] T 0ff4979d0b8b428a88c0eca25e018e86 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:17.568742 30213 consensus_peers.cc:485] T 44949ddf1eac45cfa4fc8e1c532cf7bb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:17.576166 32359 consensus_peers.cc:485] T b26b6d19699047afaee57d6f2c887664 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:17.594216 29948 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:17.604696 32753 consensus_peers.cc:485] T a46319e2fdff439e969092ead5d7e11e P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:17.618700 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:17.623672 30341 consensus_peers.cc:485] T dd2fd47ec6684cb28d2f4678d2e90e75 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:17.650390 29948 consensus_peers.cc:485] T af39c304693645269a6af59190e95cad P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:17.714648 30079 consensus_peers.cc:485] T 85cc7734181b4c89b7d50f9e81a2f583 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:17.716763 30210 consensus_peers.cc:485] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:17.718340 30341 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:17.793406 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:17.809587 32753 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:17.883214 32359 consensus_peers.cc:485] T 22bb82c5d65640128fc3271212aafb60 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:17.902017 32359 consensus_peers.cc:485] T 575e8cde863540b09a52fe590cbf2bfc P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:17.924916 30341 consensus_peers.cc:485] T 3a04e958f3d049139ff74202611af725 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:17.966913 32750 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:17.984534 32359 consensus_peers.cc:485] T 7a6a29261186429cbc05b1819999f986 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:17.994791 30344 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:17.995635 30079 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:18.037431 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:18.039064 32753 consensus_peers.cc:485] T a63e9bcd27234f2f9a9e0b268fd73486 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:18.118986 30344 consensus_peers.cc:485] T 0cc8946505474737a610de35ddf0f2f6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:18.125943 29948 consensus_peers.cc:485] T cca9e338df054a4e9af4e5bfa3fef0c9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:18.133584 30344 consensus_peers.cc:485] T c362bd01a59b4bcf85eeb59713e58b64 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:11:18.196688 14625 raft_consensus.cc:482] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f8a0cfdfdca4148a75686dc88059cf3)
I20240501 14:11:18.196836 14625 raft_consensus.cc:504] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a91029b90bf9415a89a2f70736759ad3" member_type: VOTER last_known_addr { host: "127.22.252.133" port: 46265 } } peers { permanent_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 34357 } } peers { permanent_uuid: "5f8a0cfdfdca4148a75686dc88059cf3" member_type: VOTER last_known_addr { host: "127.22.252.134" port: 33497 } } peers { permanent_uuid: "d35c16dcb506426993811bb2519d7db3" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 36457 } }
I20240501 14:11:18.197430 14625 leader_election.cc:288] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1c217469ae4478ebd49e4ed9f7352d7 (127.22.252.132:34357), 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497), d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457)
I20240501 14:11:18.197674 30407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ce3093e24347f3a4e121af452c6361" candidate_uuid: "a91029b90bf9415a89a2f70736759ad3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1c217469ae4478ebd49e4ed9f7352d7" is_pre_election: true
I20240501 14:11:18.197820 30407 raft_consensus.cc:2454] T 89ce3093e24347f3a4e121af452c6361 P d1c217469ae4478ebd49e4ed9f7352d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91029b90bf9415a89a2f70736759ad3 in term 1.
W20240501 14:11:18.198529 32359 leader_election.cc:341] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Not found: Tablet not found: 89ce3093e24347f3a4e121af452c6361
W20240501 14:11:18.198959 32359 leader_election.cc:334] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:11:18.199018 32359 leader_election.cc:302] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a91029b90bf9415a89a2f70736759ad3, d1c217469ae4478ebd49e4ed9f7352d7; no voters: 5f8a0cfdfdca4148a75686dc88059cf3, d35c16dcb506426993811bb2519d7db3
I20240501 14:11:18.199138 14625 raft_consensus.cc:2726] T 89ce3093e24347f3a4e121af452c6361 P a91029b90bf9415a89a2f70736759ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:11:18.216912 30213 consensus_peers.cc:485] T f2bc06e76e2a41429d5625ebf99e5e5c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:18.231132 29948 consensus_peers.cc:485] T ce9baa646d304268886d2cb76294c47c P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:18.238952 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:18.273543 30210 consensus_peers.cc:485] T b7cf82d1a02a4e4a864570966fc073f0 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:18.275832 32356 consensus_peers.cc:485] T 0b34bae2736747998f30b774f1d0d571 P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:18.305012 32753 consensus_peers.cc:485] T ea69585d641342b8a1340339bb2732b3 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:18.346105 30341 consensus_peers.cc:485] T 1a277d983e2a47fda9410e1fb8c31221 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:18.348073 32753 consensus_peers.cc:485] T 440f03e9113e46f6ba4f019071ebdacb P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:18.367281 29948 consensus_peers.cc:485] T 270f2887aa3d43449695c61e2c2ae1d6 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:18.393043 30082 consensus_peers.cc:485] T 8f4669cad44f4a3c9e01c4bba3c14734 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:18.393586 32753 consensus_peers.cc:485] T bd36b13ca2274fc19b9d32d202e94f83 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:18.395218 30079 consensus_peers.cc:485] T 3dccd7ef69294def8573ea17076bbdfb P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:18.407685 30213 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:18.416940 30210 consensus_peers.cc:485] T 7e337eb723974834b07aac5eacf73232 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:18.421448 30210 consensus_peers.cc:485] T 81750b74ac3f48e082e6e6d62ef8f066 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:18.497159 29951 consensus_peers.cc:485] T 7b3a90c03de64d6891dda267167d1d7b P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:18.518157 30344 consensus_peers.cc:485] T 27564c531f6147aeac4809082ffe2402 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:18.522774 32750 consensus_peers.cc:485] T 0ca8c2fa60c543dab6f30acce1958573 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:18.524457 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:18.531049 32359 consensus_peers.cc:485] T 91e27086d98443aa9acec74d6283cd75 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:18.575075 32359 consensus_peers.cc:485] T 2c5c502325fb475f8f050ccad34feb2a P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:18.625363 29951 consensus_peers.cc:485] T bf97db677b9d426486f0fc6d751d667c P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:18.694415 29948 consensus_peers.cc:485] T d1b4aad07d454290a175352f1379393d P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:18.716692 32359 consensus_peers.cc:485] T 5518112449d9448e98f5c779f1fa3248 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:18.735741 30210 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:18.767452 30213 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:18.790280 32750 consensus_peers.cc:485] T 30928cce8b6f4dd1a0e2a5480e488e24 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:18.810648 29948 consensus_peers.cc:485] T 77b0cc955c704a449d1b0333a678d3b8 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:18.815557 30213 consensus_peers.cc:485] T 9cf197259d9648b5a742dcbb74e7f89b P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:18.823484 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:18.832702 30210 consensus_peers.cc:485] T 8c579bea324147af88ea88ac30153f02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:18.836283 30210 consensus_peers.cc:485] T 75b67ccb9a334b25ae71b553a2a84a02 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:18.836441 30213 consensus_peers.cc:485] T 19f203c753ef481dafc091ea2d60f56f P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:18.853797 30082 consensus_peers.cc:485] T 7cb8f294e9704c549e1e0590b1429fe5 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:18.914429 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:19.007663 32359 consensus_peers.cc:485] T d6776209de5e405b982f863faa556727 P a91029b90bf9415a89a2f70736759ad3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:19.030779 29951 consensus_peers.cc:485] T deee2ab875274d4eaeab84e698241ca8 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:19.037535 32356 consensus_peers.cc:485] T c90d65f39c3540f48c0d85b28130e76a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:19.062410 29951 consensus_peers.cc:485] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:19.093822 29948 consensus_peers.cc:485] T 6208e4b3d25941848b2ca533edd5a7b9 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:19.112926 32750 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:19.115623 30341 consensus_peers.cc:485] T 25884a00499a4861a9df4fd3a3791b2e P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:19.176990 32750 consensus_peers.cc:485] T 06db74bb439845d4bf043c5047c65da4 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:19.232026 30344 consensus_peers.cc:485] T 50dc380460bb4f859d25ec77291faffb P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:19.236915 30344 consensus_peers.cc:485] T 28500f605e0644b69207077f57023ca6 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:19.254151 30210 consensus_peers.cc:485] T aa8ba266a8d0406bb93d6c578fe6c7cb P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:19.262032 30344 consensus_peers.cc:485] T cfc701c19cd74d0db2f90ce580230177 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:19.268807 30082 consensus_peers.cc:485] T c7df6b8d76714eebbeacc85f403d3405 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:19.271975 30079 consensus_peers.cc:485] T a3151db71f204f74a4c958093a1b1b06 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:19.279242 32750 consensus_peers.cc:485] T 27c905e41bfb4d99af6491331b588c06 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:19.289211 30344 consensus_peers.cc:485] T f7264487a83e4574a7946e53f5dc8612 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:19.291716 32750 consensus_peers.cc:485] T 7628284cb4484ee5b87173790179def0 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:19.351758 30082 consensus_peers.cc:485] T 43ecbf8c6104474fbf5eb9315f52056a P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:19.354573 30213 consensus_peers.cc:485] T bcec6adcdf4e471b86954e6d99528a22 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:19.383766 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:19.443225 32750 consensus_peers.cc:485] T f2541122e05f45c2843d06c5342c8d01 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:19.497848 30213 consensus_peers.cc:485] T 7534d5788f96488fbad5f6334f476224 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:19.498310 32356 consensus_peers.cc:485] T 081a6dc5b80c49ce8807c11eaca0439b P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:19.537566 29948 consensus_peers.cc:462] T 3110aca9b1d643e4b07d0a869015b6f7 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111)
W20240501 14:11:19.544584 32750 consensus_peers.cc:485] T bbff41345a424fce9e328b199d4baa59 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:19.550514 32753 consensus_peers.cc:485] T caebf782fa60459b9e298aa9f1074449 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:19.610646 30082 consensus_peers.cc:485] T 290ae9a8a79345f88111981d96c60f67 P d35c16dcb506426993811bb2519d7db3 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:19.626247 30341 consensus_peers.cc:485] T 7a2cb58cf4854a238d6f1c4bfdc64ed7 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:19.689157 30210 consensus_peers.cc:485] T 1e03e76474584273ae050595a2286559 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:19.723908 32356 consensus_peers.cc:485] T 712044f41112481a8cc0a88afd0a916a P a91029b90bf9415a89a2f70736759ad3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:19.786614 30344 consensus_peers.cc:485] T 9030fb3b12cd40c493bb5f30bac37090 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:19.790131 30213 consensus_peers.cc:485] T b816de53d6a74e798a3b4d8988165c54 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:19.797889 30341 consensus_peers.cc:485] T 59b8b59d67c64e2d828aaaa93647a5a4 P d1c217469ae4478ebd49e4ed9f7352d7 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:19.801925 32753 consensus_peers.cc:485] T bf0712a5e11c46a291b90bfb491b8b25 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:19.803887 30213 consensus_peers.cc:462] T 36bf7d2992034055bccfd8146449b696 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
W20240501 14:11:19.856469 30079 consensus_peers.cc:485] T f740d894942a4bb584ca5688ef823d55 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:11:19.856715 30386 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 14:11:19.859380 29993 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:11:19.859640 327 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 14:11:19.869843 32401 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:11:19.869907 30079 consensus_peers.cc:485] T fc28b2028a9246b99e8887f03180d0c3 P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:11:19.870218 30255 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240501 14:11:19.876374 30125 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:11:19.884747 30213 consensus_peers.cc:485] T 9e5188c15666488c99717f68acfeeea4 P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:19.898828 30079 consensus_peers.cc:485] T d8430bd8cb7549f0aa4ede9a3d1ffe6e P d35c16dcb506426993811bb2519d7db3 -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:19.903432 29948 consensus_peers.cc:485] T 89d3e17f11a4460fa0d0a9a0b8342a66 P 93f2518fbadf4f9991cda93afacb363e -> Peer db935501e9104de5b15c8780681d6c38 (127.22.252.135:41915): Couldn't send request to peer db935501e9104de5b15c8780681d6c38. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.135:41915: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:19.939126 30213 consensus_peers.cc:485] T 0517ff3519644b489b912f8e5a36274c P 1d0bb69611af4876a2b3b33c4766e13a -> Peer 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): Couldn't send request to peer 5f8a0cfdfdca4148a75686dc88059cf3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/rebalancer_tool-test.cc:1203: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 46
Average Replica Count | 22.875000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 1
Maximum | 3
Average | 2.904762
stderr: W20240501 14:09:32.266310 402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:32.266572 402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:32.269637 402 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:09:32.758162 402 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 14:09:33.063206 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table abd0f09508ae4b6699a526d040fd6f96: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:33.063406 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9a4b9a0a2fe64ab2a64d6679185c2a33: could not find any suitable replica to move from server 1d0bb69611af4876a2b3b33c4766e13a to server 5f8a0cfdfdca4148a75686dc88059cf3
W20240501 14:09:33.063488 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25cba3c30d2b44599355b7317baedb1c: could not find any suitable replica to move from server db935501e9104de5b15c8780681d6c38 to server 9ecd907cf5634ac98f3f54f3dbec6849
W20240501 14:09:33.063534 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b7b3f5bc4b6649c5a97ec9f02da4e967: could not find any suitable replica to move from server d1c217469ae4478ebd49e4ed9f7352d7 to server db935501e9104de5b15c8780681d6c38
W20240501 14:09:33.063597 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 40bc333ce8c9447da3fd22a75d02001d: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server db935501e9104de5b15c8780681d6c38
W20240501 14:09:33.063647 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 979bbb7a065445c89d4cf2f3e28ed0d9: could not find any suitable replica to move from server d1c217469ae4478ebd49e4ed9f7352d7 to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:33.063771 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table abd0f09508ae4b6699a526d040fd6f96: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server 5f8a0cfdfdca4148a75686dc88059cf3
W20240501 14:09:33.063814 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4fa8cc9c35c649efb9c70f46be41dd50: could not find any suitable replica to move from server d1c217469ae4478ebd49e4ed9f7352d7 to server db935501e9104de5b15c8780681d6c38
W20240501 14:09:33.063912 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e756cf3e07134acfb79003690692e338: could not find any suitable replica to move from server 93f2518fbadf4f9991cda93afacb363e to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:33.063977 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9a4b9a0a2fe64ab2a64d6679185c2a33: could not find any suitable replica to move from server 1d0bb69611af4876a2b3b33c4766e13a to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:33.064028 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 422856ed458a4c52950e8e3e49019a55: could not find any suitable replica to move from server d1c217469ae4478ebd49e4ed9f7352d7 to server 5f8a0cfdfdca4148a75686dc88059cf3
W20240501 14:09:33.064074 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 440878c63b0f4706b34fd42316991270: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server 5f8a0cfdfdca4148a75686dc88059cf3
W20240501 14:09:33.064111 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25cba3c30d2b44599355b7317baedb1c: could not find any suitable replica to move from server a91029b90bf9415a89a2f70736759ad3 to server d35c16dcb506426993811bb2519d7db3
W20240501 14:09:33.064224 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8f3109027b0c482d89eb86da7f7ed641: could not find any suitable replica to move from server 1d0bb69611af4876a2b3b33c4766e13a to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:33.064344 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 86d7db413f934f379c4369701c5c13cd: could not find any suitable replica to move from server 93f2518fbadf4f9991cda93afacb363e to server db935501e9104de5b15c8780681d6c38
W20240501 14:09:33.064433 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df055757a36d4b9f89886608b024f0b8: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server db935501e9104de5b15c8780681d6c38
W20240501 14:09:33.064484 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2f015d5186014d24b820e1f454091285: could not find any suitable replica to move from server d1c217469ae4478ebd49e4ed9f7352d7 to server db935501e9104de5b15c8780681d6c38
W20240501 14:09:33.064530 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a61705ba694e47f3818ac35bcffaee89: could not find any suitable replica to move from server 93f2518fbadf4f9991cda93afacb363e to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:33.064591 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a49ff4b9b5fe43dea63424b7e0856b72: could not find any suitable replica to move from server d1c217469ae4478ebd49e4ed9f7352d7 to server 9ecd907cf5634ac98f3f54f3dbec6849
W20240501 14:09:33.064697 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 00bf2d6ad5af4e6480096d5ceb4b1bb8: could not find any suitable replica to move from server 93f2518fbadf4f9991cda93afacb363e to server 5f8a0cfdfdca4148a75686dc88059cf3
W20240501 14:09:33.064769 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df0504ee71e643e3a5ec9a20e1037981: could not find any suitable replica to move from server 1d0bb69611af4876a2b3b33c4766e13a to server 9ecd907cf5634ac98f3f54f3dbec6849
W20240501 14:09:33.064821 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8f3109027b0c482d89eb86da7f7ed641: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server 5f8a0cfdfdca4148a75686dc88059cf3
W20240501 14:09:33.064868 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0ceb652d71d942b2a6e0ef7686e7635b: could not find any suitable replica to move from server 93f2518fbadf4f9991cda93afacb363e to server 5f8a0cfdfdca4148a75686dc88059cf3
W20240501 14:09:33.064916 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 86d7db413f934f379c4369701c5c13cd: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server 9ecd907cf5634ac98f3f54f3dbec6849
W20240501 14:09:33.064971 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b37dbfc47a574f218dd59bef604c1bc1: could not find any suitable replica to move from server d1c217469ae4478ebd49e4ed9f7352d7 to server db935501e9104de5b15c8780681d6c38
W20240501 14:09:33.065042 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4fa8cc9c35c649efb9c70f46be41dd50: could not find any suitable replica to move from server 93f2518fbadf4f9991cda93afacb363e to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:33.065088 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df055757a36d4b9f89886608b024f0b8: could not find any suitable replica to move from server d1c217469ae4478ebd49e4ed9f7352d7 to server db935501e9104de5b15c8780681d6c38
W20240501 14:09:33.065135 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2f015d5186014d24b820e1f454091285: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:33.065184 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e756cf3e07134acfb79003690692e338: could not find any suitable replica to move from server d1c217469ae4478ebd49e4ed9f7352d7 to server db935501e9104de5b15c8780681d6c38
W20240501 14:09:33.065232 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a49ff4b9b5fe43dea63424b7e0856b72: could not find any suitable replica to move from server 93f2518fbadf4f9991cda93afacb363e to server 9ecd907cf5634ac98f3f54f3dbec6849
W20240501 14:09:33.065279 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b7b3f5bc4b6649c5a97ec9f02da4e967: could not find any suitable replica to move from server 1d0bb69611af4876a2b3b33c4766e13a to server 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:09:33.096609 402 rebalancer_tool.cc:1074] tablet af39c304693645269a6af59190e95cad: '1d0bb69611af4876a2b3b33c4766e13a' -> '9ecd907cf5634ac98f3f54f3dbec6849' move scheduled
I20240501 14:09:33.127008 402 rebalancer_tool.cc:1074] tablet 796cef2485c94b76a5f3e1944b73dcf6: '93f2518fbadf4f9991cda93afacb363e' -> 'a91029b90bf9415a89a2f70736759ad3' move scheduled
I20240501 14:09:33.188491 402 rebalancer_tool.cc:1074] tablet 8d5e1afe21654080b5aa3eb877e8f1d9: 'd1c217469ae4478ebd49e4ed9f7352d7' -> '5f8a0cfdfdca4148a75686dc88059cf3' move scheduled
I20240501 14:09:33.244102 402 rebalancer_tool.cc:1074] tablet dd2fd47ec6684cb28d2f4678d2e90e75: 'd35c16dcb506426993811bb2519d7db3' -> 'db935501e9104de5b15c8780681d6c38' move scheduled
I20240501 14:09:33.287655 402 rebalancer_tool.cc:1074] tablet 940c8f54339f4a32be2942682a41c685: '1d0bb69611af4876a2b3b33c4766e13a' -> '9ecd907cf5634ac98f3f54f3dbec6849' move scheduled
I20240501 14:09:33.324936 402 rebalancer_tool.cc:1074] tablet 4f5314fbd59c4443974a9c304ba9b1a7: '93f2518fbadf4f9991cda93afacb363e' -> 'a91029b90bf9415a89a2f70736759ad3' move scheduled
I20240501 14:09:33.360637 402 rebalancer_tool.cc:1074] tablet 5da2beba68ba414281025fcee91ca0d0: 'd1c217469ae4478ebd49e4ed9f7352d7' -> '5f8a0cfdfdca4148a75686dc88059cf3' move scheduled
I20240501 14:09:33.404471 402 rebalancer_tool.cc:1074] tablet 3a04e958f3d049139ff74202611af725: 'd35c16dcb506426993811bb2519d7db3' -> 'db935501e9104de5b15c8780681d6c38' move scheduled
I20240501 14:09:33.452616 402 rebalancer_tool.cc:1074] tablet 8d87b2a4f6764e20afbfb43681f58467: '1d0bb69611af4876a2b3b33c4766e13a' -> '9ecd907cf5634ac98f3f54f3dbec6849' move scheduled
I20240501 14:09:33.453977 402 rebalancer_tool.cc:1084] tablet 048651987f8d4311b8b6bfb73b60ee90: '93f2518fbadf4f9991cda93afacb363e' -> 'a91029b90bf9415a89a2f70736759ad3' move ignored: Not found: 048651987f8d4311b8b6bfb73b60ee90: tablet not found
I20240501 14:09:33.454064 402 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:09:34.058148 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 979bbb7a065445c89d4cf2f3e28ed0d9: could not find any suitable replica to move from server 93f2518fbadf4f9991cda93afacb363e to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:34.058436 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e53010e4736c42a899edac137f9182cb: could not find any suitable replica to move from server a91029b90bf9415a89a2f70736759ad3 to server d1c217469ae4478ebd49e4ed9f7352d7
W20240501 14:09:34.058483 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a61705ba694e47f3818ac35bcffaee89: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:34.058513 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df0504ee71e643e3a5ec9a20e1037981: could not find any suitable replica to move from server d1c217469ae4478ebd49e4ed9f7352d7 to server 5f8a0cfdfdca4148a75686dc88059cf3
W20240501 14:09:34.058539 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 979bbb7a065445c89d4cf2f3e28ed0d9: could not find any suitable replica to move from server d1c217469ae4478ebd49e4ed9f7352d7 to server 9ecd907cf5634ac98f3f54f3dbec6849
W20240501 14:09:34.058566 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4fa8cc9c35c649efb9c70f46be41dd50: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:34.058701 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 86d7db413f934f379c4369701c5c13cd: could not find any suitable replica to move from server 1d0bb69611af4876a2b3b33c4766e13a to server db935501e9104de5b15c8780681d6c38
W20240501 14:09:34.058760 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e53010e4736c42a899edac137f9182cb: could not find any suitable replica to move from server 9ecd907cf5634ac98f3f54f3dbec6849 to server d35c16dcb506426993811bb2519d7db3
W20240501 14:09:34.058789 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8f3109027b0c482d89eb86da7f7ed641: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:34.058828 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 979bbb7a065445c89d4cf2f3e28ed0d9: could not find any suitable replica to move from server 93f2518fbadf4f9991cda93afacb363e to server db935501e9104de5b15c8780681d6c38
W20240501 14:09:34.058867 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a49ff4b9b5fe43dea63424b7e0856b72: could not find any suitable replica to move from server d1c217469ae4478ebd49e4ed9f7352d7 to server 9ecd907cf5634ac98f3f54f3dbec6849
W20240501 14:09:34.058897 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9a4b9a0a2fe64ab2a64d6679185c2a33: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server 9ecd907cf5634ac98f3f54f3dbec6849
W20240501 14:09:34.058928 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 440878c63b0f4706b34fd42316991270: could not find any suitable replica to move from server 1d0bb69611af4876a2b3b33c4766e13a to server 9ecd907cf5634ac98f3f54f3dbec6849
W20240501 14:09:34.058954 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b37dbfc47a574f218dd59bef604c1bc1: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:34.058996 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table abd0f09508ae4b6699a526d040fd6f96: could not find any suitable replica to move from server d1c217469ae4478ebd49e4ed9f7352d7 to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:34.059046 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e756cf3e07134acfb79003690692e338: could not find any suitable replica to move from server 93f2518fbadf4f9991cda93afacb363e to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:34.059075 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 422856ed458a4c52950e8e3e49019a55: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server db935501e9104de5b15c8780681d6c38
W20240501 14:09:34.059118 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df055757a36d4b9f89886608b024f0b8: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server db935501e9104de5b15c8780681d6c38
W20240501 14:09:34.059167 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b7b3f5bc4b6649c5a97ec9f02da4e967: could not find any suitable replica to move from server 1d0bb69611af4876a2b3b33c4766e13a to server db935501e9104de5b15c8780681d6c38
W20240501 14:09:34.059198 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e53010e4736c42a899edac137f9182cb: could not find any suitable replica to move from server 5f8a0cfdfdca4148a75686dc88059cf3 to server 1d0bb69611af4876a2b3b33c4766e13a
W20240501 14:09:34.059226 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8f3109027b0c482d89eb86da7f7ed641: could not find any suitable replica to move from server 1d0bb69611af4876a2b3b33c4766e13a to server 5f8a0cfdfdca4148a75686dc88059cf3
W20240501 14:09:34.059257 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df0504ee71e643e3a5ec9a20e1037981: could not find any suitable replica to move from server 93f2518fbadf4f9991cda93afacb363e to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:34.059283 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 979bbb7a065445c89d4cf2f3e28ed0d9: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server 5f8a0cfdfdca4148a75686dc88059cf3
W20240501 14:09:34.059322 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 00bf2d6ad5af4e6480096d5ceb4b1bb8: could not find any suitable replica to move from server d1c217469ae4478ebd49e4ed9f7352d7 to server db935501e9104de5b15c8780681d6c38
W20240501 14:09:34.059352 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0ceb652d71d942b2a6e0ef7686e7635b: could not find any suitable replica to move from server 1d0bb69611af4876a2b3b33c4766e13a to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:34.059381 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 40bc333ce8c9447da3fd22a75d02001d: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server 9ecd907cf5634ac98f3f54f3dbec6849
W20240501 14:09:34.059407 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a61705ba694e47f3818ac35bcffaee89: could not find any suitable replica to move from server 93f2518fbadf4f9991cda93afacb363e to server 9ecd907cf5634ac98f3f54f3dbec6849
W20240501 14:09:34.059437 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9a4b9a0a2fe64ab2a64d6679185c2a33: could not find any suitable replica to move from server 1d0bb69611af4876a2b3b33c4766e13a to server 5f8a0cfdfdca4148a75686dc88059cf3
W20240501 14:09:34.059479 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4fa8cc9c35c649efb9c70f46be41dd50: could not find any suitable replica to move from server 93f2518fbadf4f9991cda93afacb363e to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:34.059509 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 440878c63b0f4706b34fd42316991270: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server 9ecd907cf5634ac98f3f54f3dbec6849
W20240501 14:09:34.059538 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table abd0f09508ae4b6699a526d040fd6f96: could not find any suitable replica to move from server 1d0bb69611af4876a2b3b33c4766e13a to server db935501e9104de5b15c8780681d6c38
W20240501 14:09:34.059567 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e756cf3e07134acfb79003690692e338: could not find any suitable replica to move from server d1c217469ae4478ebd49e4ed9f7352d7 to server 5f8a0cfdfdca4148a75686dc88059cf3
I20240501 14:09:34.087759 402 rebalancer_tool.cc:1074] tablet 354658866eda4a2da6a2871fe0a2769d: '93f2518fbadf4f9991cda93afacb363e' -> 'db935501e9104de5b15c8780681d6c38' move scheduled
I20240501 14:09:34.114966 402 rebalancer_tool.cc:1074] tablet f4b8334e04994f5f8dc96eb91b944a47: 'd1c217469ae4478ebd49e4ed9f7352d7' -> '5f8a0cfdfdca4148a75686dc88059cf3' move scheduled
I20240501 14:09:34.147019 402 rebalancer_tool.cc:1074] tablet 1698ca3ec26741bb9b3131fe67c9682b: 'd35c16dcb506426993811bb2519d7db3' -> 'a91029b90bf9415a89a2f70736759ad3' move scheduled
I20240501 14:09:34.186934 402 rebalancer_tool.cc:1074] tablet 3b6a526c2ed84e1780ec392ef39d09b0: '1d0bb69611af4876a2b3b33c4766e13a' -> '9ecd907cf5634ac98f3f54f3dbec6849' move scheduled
I20240501 14:09:34.221150 402 rebalancer_tool.cc:1074] tablet fb466c8cb5a74897a884ac6012cc3734: '93f2518fbadf4f9991cda93afacb363e' -> 'db935501e9104de5b15c8780681d6c38' move scheduled
I20240501 14:09:34.245991 402 rebalancer_tool.cc:1074] tablet 5518112449d9448e98f5c779f1fa3248: 'd1c217469ae4478ebd49e4ed9f7352d7' -> '5f8a0cfdfdca4148a75686dc88059cf3' move scheduled
I20240501 14:09:34.278520 402 rebalancer_tool.cc:1074] tablet fb36e198bcf34ef68a9595d7cbe93e80: 'd35c16dcb506426993811bb2519d7db3' -> 'a91029b90bf9415a89a2f70736759ad3' move scheduled
I20240501 14:09:34.302099 402 rebalancer_tool.cc:1074] tablet 18918995fb03421b9dfb4d05369c1ce8: '1d0bb69611af4876a2b3b33c4766e13a' -> '9ecd907cf5634ac98f3f54f3dbec6849' move scheduled
I20240501 14:09:34.331373 402 rebalancer_tool.cc:1074] tablet 205a0fda164d46a29a031070f1d0c369: '93f2518fbadf4f9991cda93afacb363e' -> '5f8a0cfdfdca4148a75686dc88059cf3' move scheduled
I20240501 14:09:34.356323 402 rebalancer_tool.cc:1074] tablet fc28b2028a9246b99e8887f03180d0c3: 'd1c217469ae4478ebd49e4ed9f7352d7' -> 'db935501e9104de5b15c8780681d6c38' move scheduled
I20240501 14:09:34.388039 402 rebalancer_tool.cc:1074] tablet 53c51f3845c94f47b043621d98f70229: 'd35c16dcb506426993811bb2519d7db3' -> 'a91029b90bf9415a89a2f70736759ad3' move scheduled
I20240501 14:09:34.927117 402 rebalancer_tool.cc:1123] tablet 4f5314fbd59c4443974a9c304ba9b1a7: '93f2518fbadf4f9991cda93afacb363e' -> 'a91029b90bf9415a89a2f70736759ad3' move is abandoned: Not found: 4f5314fbd59c4443974a9c304ba9b1a7: tablet not found
I20240501 14:09:36.233006 402 rebalancer_tool.cc:1123] tablet 8d5e1afe21654080b5aa3eb877e8f1d9: 'd1c217469ae4478ebd49e4ed9f7352d7' -> '5f8a0cfdfdca4148a75686dc88059cf3' move is abandoned: Not found: 8d5e1afe21654080b5aa3eb877e8f1d9: tablet not found
I20240501 14:09:36.233700 402 rebalancer_tool.cc:1123] tablet 5da2beba68ba414281025fcee91ca0d0: 'd1c217469ae4478ebd49e4ed9f7352d7' -> '5f8a0cfdfdca4148a75686dc88059cf3' move is abandoned: Not found: 5da2beba68ba414281025fcee91ca0d0: tablet not found
I20240501 14:09:36.762303 402 rebalancer_tool.cc:1123] tablet 354658866eda4a2da6a2871fe0a2769d: '93f2518fbadf4f9991cda93afacb363e' -> 'db935501e9104de5b15c8780681d6c38' move is abandoned: Not found: 354658866eda4a2da6a2871fe0a2769d: tablet not found
I20240501 14:09:37.382686 402 rebalancer_tool.cc:1123] tablet fb466c8cb5a74897a884ac6012cc3734: '93f2518fbadf4f9991cda93afacb363e' -> 'db935501e9104de5b15c8780681d6c38' move is abandoned: Not found: fb466c8cb5a74897a884ac6012cc3734: tablet not found
I20240501 14:09:37.383415 402 rebalancer_tool.cc:1123] tablet f4b8334e04994f5f8dc96eb91b944a47: 'd1c217469ae4478ebd49e4ed9f7352d7' -> '5f8a0cfdfdca4148a75686dc88059cf3' move is abandoned: Not found: f4b8334e04994f5f8dc96eb91b944a47: tablet not found
I20240501 14:09:37.384074 402 rebalancer_tool.cc:1123] tablet 205a0fda164d46a29a031070f1d0c369: '93f2518fbadf4f9991cda93afacb363e' -> '5f8a0cfdfdca4148a75686dc88059cf3' move is abandoned: Not found: 205a0fda164d46a29a031070f1d0c369: tablet not found
I20240501 14:09:38.086342 402 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:09:38.317813 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9a4b9a0a2fe64ab2a64d6679185c2a33: could not find any suitable replica to move from server 1d0bb69611af4876a2b3b33c4766e13a to server 9ecd907cf5634ac98f3f54f3dbec6849
W20240501 14:09:38.318010 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table abd0f09508ae4b6699a526d040fd6f96: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server 5f8a0cfdfdca4148a75686dc88059cf3
W20240501 14:09:38.318084 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b7b3f5bc4b6649c5a97ec9f02da4e967: could not find any suitable replica to move from server 1d0bb69611af4876a2b3b33c4766e13a to server db935501e9104de5b15c8780681d6c38
W20240501 14:09:38.318202 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 00bf2d6ad5af4e6480096d5ceb4b1bb8: could not find any suitable replica to move from server 1d0bb69611af4876a2b3b33c4766e13a to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:38.318333 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 440878c63b0f4706b34fd42316991270: could not find any suitable replica to move from server 1d0bb69611af4876a2b3b33c4766e13a to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:38.318409 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8ff8f244c547410f853c771c11ecdd1e: could not find any suitable replica to move from server a91029b90bf9415a89a2f70736759ad3 to server 93f2518fbadf4f9991cda93afacb363e
W20240501 14:09:38.318506 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 422856ed458a4c52950e8e3e49019a55: could not find any suitable replica to move from server d1c217469ae4478ebd49e4ed9f7352d7 to server 5f8a0cfdfdca4148a75686dc88059cf3
W20240501 14:09:38.318548 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8f3109027b0c482d89eb86da7f7ed641: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:38.318596 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d23b9f10156c4074a707699cf10e3b55: could not find any suitable replica to move from server a91029b90bf9415a89a2f70736759ad3 to server 1d0bb69611af4876a2b3b33c4766e13a
W20240501 14:09:38.318710 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c8e44abe11554acda26e71b8a11e2e0e: could not find any suitable replica to move from server 5f8a0cfdfdca4148a75686dc88059cf3 to server d1c217469ae4478ebd49e4ed9f7352d7
W20240501 14:09:38.318773 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0ceb652d71d942b2a6e0ef7686e7635b: could not find any suitable replica to move from server 1d0bb69611af4876a2b3b33c4766e13a to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:38.318817 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df055757a36d4b9f89886608b024f0b8: could not find any suitable replica to move from server d1c217469ae4478ebd49e4ed9f7352d7 to server 9ecd907cf5634ac98f3f54f3dbec6849
W20240501 14:09:38.318859 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table abd0f09508ae4b6699a526d040fd6f96: could not find any suitable replica to move from server d1c217469ae4478ebd49e4ed9f7352d7 to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:38.318953 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df0504ee71e643e3a5ec9a20e1037981: could not find any suitable replica to move from server 93f2518fbadf4f9991cda93afacb363e to server db935501e9104de5b15c8780681d6c38
W20240501 14:09:38.318994 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c8e44abe11554acda26e71b8a11e2e0e: could not find any suitable replica to move from server a91029b90bf9415a89a2f70736759ad3 to server d35c16dcb506426993811bb2519d7db3
W20240501 14:09:38.319075 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 440878c63b0f4706b34fd42316991270: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server db935501e9104de5b15c8780681d6c38
I20240501 14:09:38.344749 402 rebalancer_tool.cc:1074] tablet c5d406c64b2e432a88e8c675b91dd144: '93f2518fbadf4f9991cda93afacb363e' -> '5f8a0cfdfdca4148a75686dc88059cf3' move scheduled
I20240501 14:09:38.370025 402 rebalancer_tool.cc:1074] tablet 44949ddf1eac45cfa4fc8e1c532cf7bb: 'd1c217469ae4478ebd49e4ed9f7352d7' -> '5f8a0cfdfdca4148a75686dc88059cf3' move scheduled
I20240501 14:09:38.395418 402 rebalancer_tool.cc:1074] tablet f2541122e05f45c2843d06c5342c8d01: '93f2518fbadf4f9991cda93afacb363e' -> 'db935501e9104de5b15c8780681d6c38' move scheduled
I20240501 14:09:38.422153 402 rebalancer_tool.cc:1074] tablet 35dd2c9d436448c482ef0b3fac3a3b64: '5f8a0cfdfdca4148a75686dc88059cf3' -> 'd1c217469ae4478ebd49e4ed9f7352d7' move scheduled
I20240501 14:09:38.453362 402 rebalancer_tool.cc:1074] tablet 75b67ccb9a334b25ae71b553a2a84a02: '93f2518fbadf4f9991cda93afacb363e' -> 'db935501e9104de5b15c8780681d6c38' move scheduled
I20240501 14:09:38.476327 402 rebalancer_tool.cc:1074] tablet 66471362a1d24a8dab53f9cad512a73a: 'a91029b90bf9415a89a2f70736759ad3' -> 'd1c217469ae4478ebd49e4ed9f7352d7' move scheduled
I20240501 14:09:38.498944 402 rebalancer_tool.cc:1074] tablet 0517ff3519644b489b912f8e5a36274c: '93f2518fbadf4f9991cda93afacb363e' -> '5f8a0cfdfdca4148a75686dc88059cf3' move scheduled
E20240501 14:09:40.553488 3270 ksck_remote.cc:630] unable to open table rebalancer_dynamic_table_new_name: Not found: the table does not exist: table_name: "rebalancer_dynamic_table_new_name"
I20240501 14:09:40.565188 402 rebalancer_tool.cc:1136] tablet dd2fd47ec6684cb28d2f4678d2e90e75: 'd35c16dcb506426993811bb2519d7db3' -> 'db935501e9104de5b15c8780681d6c38' move completed: OK
I20240501 14:09:40.570453 402 rebalancer_tool.cc:1136] tablet 940c8f54339f4a32be2942682a41c685: '1d0bb69611af4876a2b3b33c4766e13a' -> '9ecd907cf5634ac98f3f54f3dbec6849' move completed: OK
I20240501 14:09:40.575496 402 rebalancer_tool.cc:1136] tablet af39c304693645269a6af59190e95cad: '1d0bb69611af4876a2b3b33c4766e13a' -> '9ecd907cf5634ac98f3f54f3dbec6849' move completed: OK
I20240501 14:09:40.576262 402 rebalancer_tool.cc:1123] tablet 796cef2485c94b76a5f3e1944b73dcf6: '93f2518fbadf4f9991cda93afacb363e' -> 'a91029b90bf9415a89a2f70736759ad3' move is abandoned: Not found: 796cef2485c94b76a5f3e1944b73dcf6: tablet not found
I20240501 14:09:40.920518 402 rebalancer_tool.cc:1136] tablet 3a04e958f3d049139ff74202611af725: 'd35c16dcb506426993811bb2519d7db3' -> 'db935501e9104de5b15c8780681d6c38' move completed: OK
I20240501 14:09:41.139405 402 rebalancer_tool.cc:1136] tablet fc28b2028a9246b99e8887f03180d0c3: 'd1c217469ae4478ebd49e4ed9f7352d7' -> 'db935501e9104de5b15c8780681d6c38' move completed: OK
I20240501 14:09:41.152158 402 rebalancer_tool.cc:1136] tablet 8d87b2a4f6764e20afbfb43681f58467: '1d0bb69611af4876a2b3b33c4766e13a' -> '9ecd907cf5634ac98f3f54f3dbec6849' move completed: OK
I20240501 14:09:41.157959 402 rebalancer_tool.cc:1136] tablet 1698ca3ec26741bb9b3131fe67c9682b: 'd35c16dcb506426993811bb2519d7db3' -> 'a91029b90bf9415a89a2f70736759ad3' move completed: OK
I20240501 14:09:41.164439 402 rebalancer_tool.cc:1136] tablet fb36e198bcf34ef68a9595d7cbe93e80: 'd35c16dcb506426993811bb2519d7db3' -> 'a91029b90bf9415a89a2f70736759ad3' move completed: OK
I20240501 14:09:41.207644 402 rebalancer_tool.cc:1074] tablet c362bd01a59b4bcf85eeb59713e58b64: 'd35c16dcb506426993811bb2519d7db3' -> 'db935501e9104de5b15c8780681d6c38' move scheduled
I20240501 14:09:41.237303 402 rebalancer_tool.cc:1074] tablet ee998c3fcf3449d5845ac1c950f326d7: '1d0bb69611af4876a2b3b33c4766e13a' -> '9ecd907cf5634ac98f3f54f3dbec6849' move scheduled
I20240501 14:09:41.263617 402 rebalancer_tool.cc:1074] tablet 4a5869fb581b4939ad2207ab9200d674: 'd35c16dcb506426993811bb2519d7db3' -> 'a91029b90bf9415a89a2f70736759ad3' move scheduled
I20240501 14:09:41.290458 402 rebalancer_tool.cc:1074] tablet 588dd26e1820409a9dfccd8ea6a30f49: 'db935501e9104de5b15c8780681d6c38' -> '1d0bb69611af4876a2b3b33c4766e13a' move scheduled
I20240501 14:09:41.324980 402 rebalancer_tool.cc:1074] tablet 776d6722a14345f490f69f90d80d6a6a: '9ecd907cf5634ac98f3f54f3dbec6849' -> 'd35c16dcb506426993811bb2519d7db3' move scheduled
I20240501 14:09:41.373457 402 rebalancer_tool.cc:1074] tablet 7e1895c0656d49c09266cdbd231497cb: 'd1c217469ae4478ebd49e4ed9f7352d7' -> '9ecd907cf5634ac98f3f54f3dbec6849' move scheduled
I20240501 14:09:41.399865 402 rebalancer_tool.cc:1074] tablet 0ca8c2fa60c543dab6f30acce1958573: 'a91029b90bf9415a89a2f70736759ad3' -> '1d0bb69611af4876a2b3b33c4766e13a' move scheduled
I20240501 14:09:41.446379 402 rebalancer_tool.cc:1074] tablet 575e8cde863540b09a52fe590cbf2bfc: 'db935501e9104de5b15c8780681d6c38' -> 'd35c16dcb506426993811bb2519d7db3' move scheduled
I20240501 14:09:44.387358 402 rebalancer_tool.cc:1136] tablet 53c51f3845c94f47b043621d98f70229: 'd35c16dcb506426993811bb2519d7db3' -> 'a91029b90bf9415a89a2f70736759ad3' move completed: OK
I20240501 14:09:44.399652 402 rebalancer_tool.cc:1136] tablet 18918995fb03421b9dfb4d05369c1ce8: '1d0bb69611af4876a2b3b33c4766e13a' -> '9ecd907cf5634ac98f3f54f3dbec6849' move completed: OK
I20240501 14:09:44.405038 402 rebalancer_tool.cc:1136] tablet 75b67ccb9a334b25ae71b553a2a84a02: '93f2518fbadf4f9991cda93afacb363e' -> 'db935501e9104de5b15c8780681d6c38' move completed: OK
I20240501 14:09:44.409893 402 rebalancer_tool.cc:1136] tablet 5518112449d9448e98f5c779f1fa3248: 'd1c217469ae4478ebd49e4ed9f7352d7' -> '5f8a0cfdfdca4148a75686dc88059cf3' move completed: OK
I20240501 14:09:44.419745 402 rebalancer_tool.cc:1136] tablet 35dd2c9d436448c482ef0b3fac3a3b64: '5f8a0cfdfdca4148a75686dc88059cf3' -> 'd1c217469ae4478ebd49e4ed9f7352d7' move completed: OK
I20240501 14:09:44.867599 402 rebalancer_tool.cc:1136] tablet f2541122e05f45c2843d06c5342c8d01: '93f2518fbadf4f9991cda93afacb363e' -> 'db935501e9104de5b15c8780681d6c38' move completed: OK
I20240501 14:09:44.874537 402 rebalancer_tool.cc:1136] tablet 3b6a526c2ed84e1780ec392ef39d09b0: '1d0bb69611af4876a2b3b33c4766e13a' -> '9ecd907cf5634ac98f3f54f3dbec6849' move completed: OK
I20240501 14:09:44.899798 402 rebalancer_tool.cc:1074] tablet c7df6b8d76714eebbeacc85f403d3405: 'db935501e9104de5b15c8780681d6c38' -> '5f8a0cfdfdca4148a75686dc88059cf3' move scheduled
I20240501 14:09:44.923521 402 rebalancer_tool.cc:1074] tablet 7a6a29261186429cbc05b1819999f986: '9ecd907cf5634ac98f3f54f3dbec6849' -> 'd1c217469ae4478ebd49e4ed9f7352d7' move scheduled
I20240501 14:09:44.954941 402 rebalancer_tool.cc:1074] tablet 15c98173a71d4f73b8f6af24ad01d7c4: '1d0bb69611af4876a2b3b33c4766e13a' -> 'a91029b90bf9415a89a2f70736759ad3' move scheduled
I20240501 14:09:44.987419 402 rebalancer_tool.cc:1074] tablet 2c5c502325fb475f8f050ccad34feb2a: 'db935501e9104de5b15c8780681d6c38' -> 'd35c16dcb506426993811bb2519d7db3' move scheduled
I20240501 14:09:45.033366 402 rebalancer_tool.cc:1074] tablet e122adbba7c749cd93c53fa07a8a8b84: '5f8a0cfdfdca4148a75686dc88059cf3' -> '9ecd907cf5634ac98f3f54f3dbec6849' move scheduled
I20240501 14:09:46.804587 402 rebalancer_tool.cc:1136] tablet 776d6722a14345f490f69f90d80d6a6a: '9ecd907cf5634ac98f3f54f3dbec6849' -> 'd35c16dcb506426993811bb2519d7db3' move completed: OK
I20240501 14:09:46.811621 402 rebalancer_tool.cc:1136] tablet ee998c3fcf3449d5845ac1c950f326d7: '1d0bb69611af4876a2b3b33c4766e13a' -> '9ecd907cf5634ac98f3f54f3dbec6849' move completed: OK
I20240501 14:09:47.300832 402 rebalancer_tool.cc:1136] tablet 0517ff3519644b489b912f8e5a36274c: '93f2518fbadf4f9991cda93afacb363e' -> '5f8a0cfdfdca4148a75686dc88059cf3' move completed: OK
I20240501 14:09:47.531940 402 rebalancer_tool.cc:1136] tablet 66471362a1d24a8dab53f9cad512a73a: 'a91029b90bf9415a89a2f70736759ad3' -> 'd1c217469ae4478ebd49e4ed9f7352d7' move completed: OK
I20240501 14:09:47.537247 402 rebalancer_tool.cc:1136] tablet 4a5869fb581b4939ad2207ab9200d674: 'd35c16dcb506426993811bb2519d7db3' -> 'a91029b90bf9415a89a2f70736759ad3' move completed: OK
I20240501 14:09:47.542563 402 rebalancer_tool.cc:1136] tablet 44949ddf1eac45cfa4fc8e1c532cf7bb: 'd1c217469ae4478ebd49e4ed9f7352d7' -> '5f8a0cfdfdca4148a75686dc88059cf3' move completed: OK
I20240501 14:09:47.846923 402 rebalancer_tool.cc:1136] tablet 588dd26e1820409a9dfccd8ea6a30f49: 'db935501e9104de5b15c8780681d6c38' -> '1d0bb69611af4876a2b3b33c4766e13a' move completed: OK
I20240501 14:09:47.853492 402 rebalancer_tool.cc:1136] tablet c5d406c64b2e432a88e8c675b91dd144: '93f2518fbadf4f9991cda93afacb363e' -> '5f8a0cfdfdca4148a75686dc88059cf3' move completed: OK
I20240501 14:09:47.876991 402 rebalancer_tool.cc:1074] tablet deee2ab875274d4eaeab84e698241ca8: '1d0bb69611af4876a2b3b33c4766e13a' -> '5f8a0cfdfdca4148a75686dc88059cf3' move scheduled
I20240501 14:09:47.904564 402 rebalancer_tool.cc:1074] tablet 371c725533d84656b7a76c1bc981d038: 'd35c16dcb506426993811bb2519d7db3' -> '9ecd907cf5634ac98f3f54f3dbec6849' move scheduled
I20240501 14:09:47.931331 402 rebalancer_tool.cc:1074] tablet 7b3a90c03de64d6891dda267167d1d7b: 'a91029b90bf9415a89a2f70736759ad3' -> '1d0bb69611af4876a2b3b33c4766e13a' move scheduled
I20240501 14:09:47.959545 402 rebalancer_tool.cc:1074] tablet b66c71cbccfa4c96bdf2b438938bf56b: '5f8a0cfdfdca4148a75686dc88059cf3' -> 'd35c16dcb506426993811bb2519d7db3' move scheduled
I20240501 14:09:47.995065 402 rebalancer_tool.cc:1074] tablet b26b6d19699047afaee57d6f2c887664: 'db935501e9104de5b15c8780681d6c38' -> '9ecd907cf5634ac98f3f54f3dbec6849' move scheduled
I20240501 14:09:48.033442 402 rebalancer_tool.cc:1074] tablet 1e03e76474584273ae050595a2286559: 'a91029b90bf9415a89a2f70736759ad3' -> '1d0bb69611af4876a2b3b33c4766e13a' move scheduled
W20240501 14:09:48.854354 6034 meta_cache.cc:1015] Not found: LookupRpc { table: 'rebalancer_dynamic_table_new_name', partition-key: (<start>), attempt: 1 } failed: table rebalancer_dynamic_table_new_name was deleted: Table deleted at 2024-05-01 14:09:48 UTC
I20240501 14:09:50.644629 402 rebalancer_tool.cc:1136] tablet 575e8cde863540b09a52fe590cbf2bfc: 'db935501e9104de5b15c8780681d6c38' -> 'd35c16dcb506426993811bb2519d7db3' move completed: OK
I20240501 14:09:50.650936 402 rebalancer_tool.cc:1136] tablet 0ca8c2fa60c543dab6f30acce1958573: 'a91029b90bf9415a89a2f70736759ad3' -> '1d0bb69611af4876a2b3b33c4766e13a' move completed: OK
I20240501 14:09:50.658479 402 rebalancer_tool.cc:1136] tablet 7e1895c0656d49c09266cdbd231497cb: 'd1c217469ae4478ebd49e4ed9f7352d7' -> '9ecd907cf5634ac98f3f54f3dbec6849' move completed: OK
I20240501 14:09:50.664433 402 rebalancer_tool.cc:1136] tablet c362bd01a59b4bcf85eeb59713e58b64: 'd35c16dcb506426993811bb2519d7db3' -> 'db935501e9104de5b15c8780681d6c38' move completed: OK
I20240501 14:09:50.671700 402 rebalancer_tool.cc:1136] tablet c7df6b8d76714eebbeacc85f403d3405: 'db935501e9104de5b15c8780681d6c38' -> '5f8a0cfdfdca4148a75686dc88059cf3' move completed: OK
I20240501 14:09:50.681737 402 rebalancer_tool.cc:1136] tablet e122adbba7c749cd93c53fa07a8a8b84: '5f8a0cfdfdca4148a75686dc88059cf3' -> '9ecd907cf5634ac98f3f54f3dbec6849' move completed: OK
I20240501 14:09:50.704723 402 rebalancer_tool.cc:1074] tablet 529f938305214c3ba33f61e06d1acd77: '1d0bb69611af4876a2b3b33c4766e13a' -> 'db935501e9104de5b15c8780681d6c38' move scheduled
I20240501 14:09:50.733058 402 rebalancer_tool.cc:1074] tablet 19f203c753ef481dafc091ea2d60f56f: 'd35c16dcb506426993811bb2519d7db3' -> '5f8a0cfdfdca4148a75686dc88059cf3' move scheduled
I20240501 14:09:50.757540 402 rebalancer_tool.cc:1074] tablet f2bc06e76e2a41429d5625ebf99e5e5c: 'd35c16dcb506426993811bb2519d7db3' -> '5f8a0cfdfdca4148a75686dc88059cf3' move scheduled
E20240501 14:09:51.614257 6959 ksck_remote.cc:630] unable to open table rebalancer_dynamic_table_new_name: Not found: the table does not exist: table_name: "rebalancer_dynamic_table_new_name"
I20240501 14:09:52.170240 402 rebalancer_tool.cc:1136] tablet 371c725533d84656b7a76c1bc981d038: 'd35c16dcb506426993811bb2519d7db3' -> '9ecd907cf5634ac98f3f54f3dbec6849' move completed: OK
I20240501 14:09:52.178189 402 rebalancer_tool.cc:1136] tablet deee2ab875274d4eaeab84e698241ca8: '1d0bb69611af4876a2b3b33c4766e13a' -> '5f8a0cfdfdca4148a75686dc88059cf3' move completed: OK
I20240501 14:09:52.188815 402 rebalancer_tool.cc:1136] tablet 2c5c502325fb475f8f050ccad34feb2a: 'db935501e9104de5b15c8780681d6c38' -> 'd35c16dcb506426993811bb2519d7db3' move completed: OK
I20240501 14:09:52.208506 402 rebalancer_tool.cc:1136] tablet 15c98173a71d4f73b8f6af24ad01d7c4: '1d0bb69611af4876a2b3b33c4766e13a' -> 'a91029b90bf9415a89a2f70736759ad3' move completed: OK
I20240501 14:09:52.220600 402 rebalancer_tool.cc:1136] tablet 7a6a29261186429cbc05b1819999f986: '9ecd907cf5634ac98f3f54f3dbec6849' -> 'd1c217469ae4478ebd49e4ed9f7352d7' move completed: OK
I20240501 14:09:52.225960 402 rebalancer_tool.cc:1136] tablet b26b6d19699047afaee57d6f2c887664: 'db935501e9104de5b15c8780681d6c38' -> '9ecd907cf5634ac98f3f54f3dbec6849' move completed: OK
I20240501 14:09:52.251072 402 rebalancer_tool.cc:1074] tablet 21efaac3a3d643679c5abaaddb9e5a3a: '9ecd907cf5634ac98f3f54f3dbec6849' -> '1d0bb69611af4876a2b3b33c4766e13a' move scheduled
I20240501 14:09:52.276095 402 rebalancer_tool.cc:1074] tablet f740d894942a4bb584ca5688ef823d55: '1d0bb69611af4876a2b3b33c4766e13a' -> 'db935501e9104de5b15c8780681d6c38' move scheduled
I20240501 14:09:54.332427 402 rebalancer_tool.cc:1136] tablet 7b3a90c03de64d6891dda267167d1d7b: 'a91029b90bf9415a89a2f70736759ad3' -> '1d0bb69611af4876a2b3b33c4766e13a' move completed: OK
I20240501 14:09:54.338074 402 rebalancer_tool.cc:1136] tablet 1e03e76474584273ae050595a2286559: 'a91029b90bf9415a89a2f70736759ad3' -> '1d0bb69611af4876a2b3b33c4766e13a' move completed: OK
I20240501 14:09:54.366042 402 rebalancer_tool.cc:1074] tablet 3125c67fe40641c2bfd5dcac81e7237f: '1d0bb69611af4876a2b3b33c4766e13a' -> 'db935501e9104de5b15c8780681d6c38' move scheduled
I20240501 14:09:54.393957 402 rebalancer_tool.cc:1074] tablet cfc701c19cd74d0db2f90ce580230177: '1d0bb69611af4876a2b3b33c4766e13a' -> '5f8a0cfdfdca4148a75686dc88059cf3' move scheduled
W20240501 14:09:55.015393 7948 meta_cache.cc:1261] Time spent looking up entry by key: real 0.051s user 0.000s sys 0.000s
I20240501 14:09:55.449617 402 rebalancer_tool.cc:1136] tablet 19f203c753ef481dafc091ea2d60f56f: 'd35c16dcb506426993811bb2519d7db3' -> '5f8a0cfdfdca4148a75686dc88059cf3' move completed: OK
I20240501 14:09:55.460494 402 rebalancer_tool.cc:1136] tablet 529f938305214c3ba33f61e06d1acd77: '1d0bb69611af4876a2b3b33c4766e13a' -> 'db935501e9104de5b15c8780681d6c38' move completed: OK
I20240501 14:09:55.465818 402 rebalancer_tool.cc:1136] tablet f2bc06e76e2a41429d5625ebf99e5e5c: 'd35c16dcb506426993811bb2519d7db3' -> '5f8a0cfdfdca4148a75686dc88059cf3' move completed: OK
I20240501 14:09:55.470955 402 rebalancer_tool.cc:1136] tablet b66c71cbccfa4c96bdf2b438938bf56b: '5f8a0cfdfdca4148a75686dc88059cf3' -> 'd35c16dcb506426993811bb2519d7db3' move completed: OK
I20240501 14:09:56.060249 402 rebalancer_tool.cc:1136] tablet f740d894942a4bb584ca5688ef823d55: '1d0bb69611af4876a2b3b33c4766e13a' -> 'db935501e9104de5b15c8780681d6c38' move completed: OK
I20240501 14:09:56.065485 402 rebalancer_tool.cc:1136] tablet 21efaac3a3d643679c5abaaddb9e5a3a: '9ecd907cf5634ac98f3f54f3dbec6849' -> '1d0bb69611af4876a2b3b33c4766e13a' move completed: OK
I20240501 14:09:56.071620 402 rebalancer_tool.cc:1136] tablet cfc701c19cd74d0db2f90ce580230177: '1d0bb69611af4876a2b3b33c4766e13a' -> '5f8a0cfdfdca4148a75686dc88059cf3' move completed: OK
I20240501 14:09:56.076458 402 rebalancer_tool.cc:1136] tablet 3125c67fe40641c2bfd5dcac81e7237f: '1d0bb69611af4876a2b3b33c4766e13a' -> 'db935501e9104de5b15c8780681d6c38' move completed: OK
W20240501 14:09:56.373598 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b2179656a9324d9f96ca0af825446742: could not find any suitable replica to move from server 93f2518fbadf4f9991cda93afacb363e to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:56.374399 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c280593e367426fa4c2534483cb4da4: could not find any suitable replica to move from server 9ecd907cf5634ac98f3f54f3dbec6849 to server d35c16dcb506426993811bb2519d7db3
W20240501 14:09:56.374460 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b2179656a9324d9f96ca0af825446742: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server db935501e9104de5b15c8780681d6c38
W20240501 14:09:56.374536 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8c1db6a412e9480587e107e98904d00e: could not find any suitable replica to move from server 5f8a0cfdfdca4148a75686dc88059cf3 to server d35c16dcb506426993811bb2519d7db3
W20240501 14:09:56.374625 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9ad5e8a877804817a0b33b99b908abad: could not find any suitable replica to move from server a91029b90bf9415a89a2f70736759ad3 to server db935501e9104de5b15c8780681d6c38
W20240501 14:09:56.374675 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table abd0f09508ae4b6699a526d040fd6f96: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server 5f8a0cfdfdca4148a75686dc88059cf3
W20240501 14:09:56.374768 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fa42a548a7f24eab9694087203fda9ba: could not find any suitable replica to move from server a91029b90bf9415a89a2f70736759ad3 to server d1c217469ae4478ebd49e4ed9f7352d7
W20240501 14:09:56.374895 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b0f02750933049cf9e3457c96b5a5534: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server 5f8a0cfdfdca4148a75686dc88059cf3
W20240501 14:09:56.374953 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 02ac9598abf44cafb0f816aa3165fb2c: could not find any suitable replica to move from server d35c16dcb506426993811bb2519d7db3 to server d1c217469ae4478ebd49e4ed9f7352d7
W20240501 14:09:56.375065 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c6639c36fcc54bb09704a0d7c258496b: could not find any suitable replica to move from server 5f8a0cfdfdca4148a75686dc88059cf3 to server a91029b90bf9415a89a2f70736759ad3
W20240501 14:09:56.375150 402 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fa42a548a7f24eab9694087203fda9ba: could not find any suitable replica to move from server 9ecd907cf5634ac98f3f54f3dbec6849 to server 1d0bb69611af4876a2b3b33c4766e13a
I20240501 14:09:56.408708 402 rebalancer_tool.cc:1074] tablet 14267ff763f5409998f457e14e407ed0: 'db935501e9104de5b15c8780681d6c38' -> 'd1c217469ae4478ebd49e4ed9f7352d7' move scheduled
I20240501 14:09:56.439652 402 rebalancer_tool.cc:1074] tablet dd2fd47ec6684cb28d2f4678d2e90e75: '1d0bb69611af4876a2b3b33c4766e13a' -> '9ecd907cf5634ac98f3f54f3dbec6849' move scheduled
I20240501 14:09:56.506503 402 rebalancer_tool.cc:1074] tablet a4462d7eaaa54223998369121761acd6: '5f8a0cfdfdca4148a75686dc88059cf3' -> 'd35c16dcb506426993811bb2519d7db3' move scheduled
I20240501 14:09:56.537214 402 rebalancer_tool.cc:1074] tablet c7df6b8d76714eebbeacc85f403d3405: 'a91029b90bf9415a89a2f70736759ad3' -> '93f2518fbadf4f9991cda93afacb363e' move scheduled
I20240501 14:09:56.566401 402 rebalancer_tool.cc:1074] tablet 8bb9687045d44efa888a69c7acde1105: 'db935501e9104de5b15c8780681d6c38' -> 'd1c217469ae4478ebd49e4ed9f7352d7' move scheduled
I20240501 14:09:56.593792 402 rebalancer_tool.cc:1074] tablet c362bd01a59b4bcf85eeb59713e58b64: '1d0bb69611af4876a2b3b33c4766e13a' -> '5f8a0cfdfdca4148a75686dc88059cf3' move scheduled
I20240501 14:09:56.647051 402 rebalancer_tool.cc:1074] tablet cfc701c19cd74d0db2f90ce580230177: 'd35c16dcb506426993811bb2519d7db3' -> '9ecd907cf5634ac98f3f54f3dbec6849' move scheduled
I20240501 14:09:56.680328 402 rebalancer_tool.cc:1074] tablet 33de572fa35c40109cb70ee3287829f2: 'a91029b90bf9415a89a2f70736759ad3' -> '93f2518fbadf4f9991cda93afacb363e' move scheduled
I20240501 14:09:56.710040 402 rebalancer_tool.cc:1074] tablet 5db25d87e72347f4a4410bcde440f359: 'db935501e9104de5b15c8780681d6c38' -> 'd1c217469ae4478ebd49e4ed9f7352d7' move scheduled
I20240501 14:09:56.747763 402 rebalancer_tool.cc:1074] tablet b0ef35582c4f4d5c95199e00ccac92c9: '5f8a0cfdfdca4148a75686dc88059cf3' -> '1d0bb69611af4876a2b3b33c4766e13a' move scheduled
I20240501 14:09:56.777674 402 rebalancer_tool.cc:1074] tablet 940c8f54339f4a32be2942682a41c685: 'd35c16dcb506426993811bb2519d7db3' -> 'a91029b90bf9415a89a2f70736759ad3' move scheduled
I20240501 14:09:56.815773 402 rebalancer_tool.cc:1074] tablet 6208e4b3d25941848b2ca533edd5a7b9: '9ecd907cf5634ac98f3f54f3dbec6849' -> '93f2518fbadf4f9991cda93afacb363e' move scheduled
I20240501 14:09:56.849391 402 rebalancer_tool.cc:1074] tablet af39c304693645269a6af59190e95cad: 'd1c217469ae4478ebd49e4ed9f7352d7' -> 'db935501e9104de5b15c8780681d6c38' move scheduled
I20240501 14:09:56.883697 402 rebalancer_tool.cc:1074] tablet 7e337eb723974834b07aac5eacf73232: '5f8a0cfdfdca4148a75686dc88059cf3' -> '1d0bb69611af4876a2b3b33c4766e13a' move scheduled
I20240501 14:09:56.911334 402 rebalancer_tool.cc:1074] tablet 5518112449d9448e98f5c779f1fa3248: 'd35c16dcb506426993811bb2519d7db3' -> 'a91029b90bf9415a89a2f70736759ad3' move scheduled
I20240501 14:09:56.948083 402 rebalancer_tool.cc:1074] tablet 0cccc8eec30141c1946b57d3be27000f: '9ecd907cf5634ac98f3f54f3dbec6849' -> '93f2518fbadf4f9991cda93afacb363e' move scheduled
I20240501 14:09:57.004880 402 rebalancer_tool.cc:1074] tablet d8430bd8cb7549f0aa4ede9a3d1ffe6e: 'd1c217469ae4478ebd49e4ed9f7352d7' -> '1d0bb69611af4876a2b3b33c4766e13a' move scheduled
I20240501 14:09:57.045142 402 rebalancer_tool.cc:1074] tablet bf0712a5e11c46a291b90bfb491b8b25: 'db935501e9104de5b15c8780681d6c38' -> '5f8a0cfdfdca4148a75686dc88059cf3' move scheduled
I20240501 14:09:57.079036 402 rebalancer_tool.cc:1074] tablet 28500f605e0644b69207077f57023ca6: 'a91029b90bf9415a89a2f70736759ad3' -> 'd35c16dcb506426993811bb2519d7db3' move scheduled
I20240501 14:09:57.117933 402 rebalancer_tool.cc:1074] tablet 65bc0e6ad1fa491e8e5ae225f86561f9: '9ecd907cf5634ac98f3f54f3dbec6849' -> '93f2518fbadf4f9991cda93afacb363e' move scheduled
W20240501 14:09:58.071265 9081 meta_cache.cc:1015] Not found: LookupRpc { table: 'rebalancer_dynamic_table_new_name', partition-key: (<start>), attempt: 1 } failed: table rebalancer_dynamic_table_new_name was deleted: Table deleted at 2024-05-01 14:09:58 UTC
I20240501 14:10:05.026120 402 rebalancer_tool.cc:1136] tablet bf0712a5e11c46a291b90bfb491b8b25: 'db935501e9104de5b15c8780681d6c38' -> '5f8a0cfdfdca4148a75686dc88059cf3' move completed: OK
I20240501 14:10:05.032178 402 rebalancer_tool.cc:1136] tablet d8430bd8cb7549f0aa4ede9a3d1ffe6e: 'd1c217469ae4478ebd49e4ed9f7352d7' -> '1d0bb69611af4876a2b3b33c4766e13a' move completed: OK
I20240501 14:10:05.038694 402 rebalancer_tool.cc:1136] tablet 7e337eb723974834b07aac5eacf73232: '5f8a0cfdfdca4148a75686dc88059cf3' -> '1d0bb69611af4876a2b3b33c4766e13a' move completed: OK
I20240501 14:10:05.044809 402 rebalancer_tool.cc:1136] tablet af39c304693645269a6af59190e95cad: 'd1c217469ae4478ebd49e4ed9f7352d7' -> 'db935501e9104de5b15c8780681d6c38' move completed: OK
I20240501 14:10:05.051036 402 rebalancer_tool.cc:1136] tablet 940c8f54339f4a32be2942682a41c685: 'd35c16dcb506426993811bb2519d7db3' -> 'a91029b90bf9415a89a2f70736759ad3' move completed: OK
I20240501 14:10:05.328609 402 rebalancer_tool.cc:1136] tablet b0ef35582c4f4d5c95199e00ccac92c9: '5f8a0cfdfdca4148a75686dc88059cf3' -> '1d0bb69611af4876a2b3b33c4766e13a' move completed: OK
I20240501 14:10:05.334376 402 rebalancer_tool.cc:1136] tablet 5db25d87e72347f4a4410bcde440f359: 'db935501e9104de5b15c8780681d6c38' -> 'd1c217469ae4478ebd49e4ed9f7352d7' move completed: OK
I20240501 14:10:05.340792 402 rebalancer_tool.cc:1136] tablet 33de572fa35c40109cb70ee3287829f2: 'a91029b90bf9415a89a2f70736759ad3' -> '93f2518fbadf4f9991cda93afacb363e' move completed: OK
I20240501 14:10:05.346491 402 rebalancer_tool.cc:1136] tablet 6208e4b3d25941848b2ca533edd5a7b9: '9ecd907cf5634ac98f3f54f3dbec6849' -> '93f2518fbadf4f9991cda93afacb363e' move completed: OK
I20240501 14:10:05.352296 402 rebalancer_tool.cc:1136] tablet cfc701c19cd74d0db2f90ce580230177: 'd35c16dcb506426993811bb2519d7db3' -> '9ecd907cf5634ac98f3f54f3dbec6849' move completed: OK
I20240501 14:10:05.357568 402 rebalancer_tool.cc:1136] tablet c362bd01a59b4bcf85eeb59713e58b64: '1d0bb69611af4876a2b3b33c4766e13a' -> '5f8a0cfdfdca4148a75686dc88059cf3' move completed: OK
I20240501 14:10:05.641311 402 rebalancer_tool.cc:1136] tablet c7df6b8d76714eebbeacc85f403d3405: 'a91029b90bf9415a89a2f70736759ad3' -> '93f2518fbadf4f9991cda93afacb363e' move completed: OK
I20240501 14:10:05.647612 402 rebalancer_tool.cc:1136] tablet 0cccc8eec30141c1946b57d3be27000f: '9ecd907cf5634ac98f3f54f3dbec6849' -> '93f2518fbadf4f9991cda93afacb363e' move completed: OK
I20240501 14:10:05.656016 402 rebalancer_tool.cc:1136] tablet a4462d7eaaa54223998369121761acd6: '5f8a0cfdfdca4148a75686dc88059cf3' -> 'd35c16dcb506426993811bb2519d7db3' move completed: OK
I20240501 14:10:05.663347 402 rebalancer_tool.cc:1136] tablet dd2fd47ec6684cb28d2f4678d2e90e75: '1d0bb69611af4876a2b3b33c4766e13a' -> '9ecd907cf5634ac98f3f54f3dbec6849' move completed: OK
I20240501 14:10:05.672490 402 rebalancer_tool.cc:1136] tablet 14267ff763f5409998f457e14e407ed0: 'db935501e9104de5b15c8780681d6c38' -> 'd1c217469ae4478ebd49e4ed9f7352d7' move completed: OK
I20240501 14:10:05.702720 402 rebalancer_tool.cc:1074] tablet 5025e07cbfd543e3b4722c6c69e4a6b3: '5f8a0cfdfdca4148a75686dc88059cf3' -> '1d0bb69611af4876a2b3b33c4766e13a' move scheduled
I20240501 14:10:05.731817 402 rebalancer_tool.cc:1074] tablet deee2ab875274d4eaeab84e698241ca8: '93f2518fbadf4f9991cda93afacb363e' -> 'db935501e9104de5b15c8780681d6c38' move scheduled
I20240501 14:10:05.765008 402 rebalancer_tool.cc:1074] tablet 36bf7d2992034055bccfd8146449b696: '9ecd907cf5634ac98f3f54f3dbec6849' -> '5f8a0cfdfdca4148a75686dc88059cf3' move scheduled
I20240501 14:10:05.791469 402 rebalancer_tool.cc:1074] tablet 75b67ccb9a334b25ae71b553a2a84a02: 'd1c217469ae4478ebd49e4ed9f7352d7' -> 'a91029b90bf9415a89a2f70736759ad3' move scheduled
I20240501 14:10:05.847267 402 rebalancer_tool.cc:1074] tablet d6776209de5e405b982f863faa556727: '1d0bb69611af4876a2b3b33c4766e13a' -> '93f2518fbadf4f9991cda93afacb363e' move scheduled
I20240501 14:10:05.877449 402 rebalancer_tool.cc:1074] tablet caebf782fa60459b9e298aa9f1074449: 'db935501e9104de5b15c8780681d6c38' -> 'd35c16dcb506426993811bb2519d7db3' move scheduled
I20240501 14:10:05.903474 402 rebalancer_tool.cc:1074] tablet 27564c531f6147aeac4809082ffe2402: '5f8a0cfdfdca4148a75686dc88059cf3' -> 'a91029b90bf9415a89a2f70736759ad3' move scheduled
I20240501 14:10:05.931398 402 rebalancer_tool.cc:1074] tablet ce04f87c84f54f1aaabb7cf10afe5b50: '93f2518fbadf4f9991cda93afacb363e' -> 'd1c217469ae4478ebd49e4ed9f7352d7' move scheduled
I20240501 14:10:05.956750 402 rebalancer_tool.cc:1074] tablet 89d3e17f11a4460fa0d0a9a0b8342a66: '1d0bb69611af4876a2b3b33c4766e13a' -> 'db935501e9104de5b15c8780681d6c38' move scheduled
I20240501 14:10:05.990056 402 rebalancer_tool.cc:1074] tablet eb828a6a5cd240f5acb591e7eae5e1a8: '5f8a0cfdfdca4148a75686dc88059cf3' -> 'd35c16dcb506426993811bb2519d7db3' move scheduled
I20240501 14:10:06.018887 402 rebalancer_tool.cc:1074] tablet e7ae4cac38e0454d8b6e019208553dac: 'a91029b90bf9415a89a2f70736759ad3' -> 'd1c217469ae4478ebd49e4ed9f7352d7' move scheduled
I20240501 14:10:06.050415 402 rebalancer_tool.cc:1074] tablet 42636717a44e4e3782037c48d8913307: '1d0bb69611af4876a2b3b33c4766e13a' -> '93f2518fbadf4f9991cda93afacb363e' move scheduled
I20240501 14:10:06.077785 402 rebalancer_tool.cc:1074] tablet ea69585d641342b8a1340339bb2732b3: 'db935501e9104de5b15c8780681d6c38' -> '1d0bb69611af4876a2b3b33c4766e13a' move scheduled
I20240501 14:10:06.120290 402 rebalancer_tool.cc:1074] tablet 89ce3093e24347f3a4e121af452c6361: 'd1c217469ae4478ebd49e4ed9f7352d7' -> 'd35c16dcb506426993811bb2519d7db3' move scheduled
W20240501 14:10:06.582935 11385 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:10:06.283078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.252.134:33497 (local address 127.0.0.1:33694)
0501 14:10:06.287888 (+ 4810us) negotiation.cc:107] Waiting for socket to connect
0501 14:10:06.287904 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 14:10:06.287974 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:10:06.582204 (+294230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":4731,"thread_start_us":108,"threads_started":1}
I20240501 14:10:09.567763 402 rebalancer_tool.cc:1136] tablet 65bc0e6ad1fa491e8e5ae225f86561f9: '9ecd907cf5634ac98f3f54f3dbec6849' -> '93f2518fbadf4f9991cda93afacb363e' move completed: OK
I20240501 14:10:09.573026 402 rebalancer_tool.cc:1136] tablet 28500f605e0644b69207077f57023ca6: 'a91029b90bf9415a89a2f70736759ad3' -> 'd35c16dcb506426993811bb2519d7db3' move completed: OK
I20240501 14:10:09.844417 402 rebalancer_tool.cc:1136] tablet 5518112449d9448e98f5c779f1fa3248: 'd35c16dcb506426993811bb2519d7db3' -> 'a91029b90bf9415a89a2f70736759ad3' move completed: OK
I20240501 14:10:09.850371 402 rebalancer_tool.cc:1136] tablet 8bb9687045d44efa888a69c7acde1105: 'db935501e9104de5b15c8780681d6c38' -> 'd1c217469ae4478ebd49e4ed9f7352d7' move completed: OK
I20240501 14:10:09.879029 402 rebalancer_tool.cc:1074] tablet 7628284cb4484ee5b87173790179def0: 'a91029b90bf9415a89a2f70736759ad3' -> 'd35c16dcb506426993811bb2519d7db3' move scheduled
I20240501 14:10:11.110940 402 rebalancer_tool.cc:1074] tablet 440f03e9113e46f6ba4f019071ebdacb: '5f8a0cfdfdca4148a75686dc88059cf3' -> '93f2518fbadf4f9991cda93afacb363e' move scheduled
I20240501 14:10:11.147073 402 rebalancer_tool.cc:1074] tablet f726c6a4929541c9b8ba7e29b757bb89: 'db935501e9104de5b15c8780681d6c38' -> 'a91029b90bf9415a89a2f70736759ad3' move scheduled
I20240501 14:10:11.662683 402 rebalancer_tool.cc:1123] tablet 89ce3093e24347f3a4e121af452c6361: 'd1c217469ae4478ebd49e4ed9f7352d7' -> 'd35c16dcb506426993811bb2519d7db3' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.22.252.134:33497: connect: Connection refused (error 111)
I20240501 14:10:12.112993 402 rebalancer_tool.cc:1136] tablet e7ae4cac38e0454d8b6e019208553dac: 'a91029b90bf9415a89a2f70736759ad3' -> 'd1c217469ae4478ebd49e4ed9f7352d7' move completed: OK
I20240501 14:10:12.336872 402 rebalancer_tool.cc:1136] tablet ce04f87c84f54f1aaabb7cf10afe5b50: '93f2518fbadf4f9991cda93afacb363e' -> 'd1c217469ae4478ebd49e4ed9f7352d7' move completed: OK
I20240501 14:10:12.547784 402 rebalancer_tool.cc:1136] tablet 89d3e17f11a4460fa0d0a9a0b8342a66: '1d0bb69611af4876a2b3b33c4766e13a' -> 'db935501e9104de5b15c8780681d6c38' move completed: OK
I20240501 14:10:14.095774 402 rebalancer_tool.cc:1136] tablet 75b67ccb9a334b25ae71b553a2a84a02: 'd1c217469ae4478ebd49e4ed9f7352d7' -> 'a91029b90bf9415a89a2f70736759ad3' move completed: OK
I20240501 14:10:14.123369 402 rebalancer_tool.cc:1074] tablet c90d65f39c3540f48c0d85b28130e76a: '93f2518fbadf4f9991cda93afacb363e' -> 'd1c217469ae4478ebd49e4ed9f7352d7' move scheduled
I20240501 14:10:15.159911 402 rebalancer_tool.cc:1074] tablet b26b6d19699047afaee57d6f2c887664: 'a91029b90bf9415a89a2f70736759ad3' -> 'd35c16dcb506426993811bb2519d7db3' move scheduled
I20240501 14:10:16.087638 402 rebalancer_tool.cc:1074] tablet f2bc06e76e2a41429d5625ebf99e5e5c: '1d0bb69611af4876a2b3b33c4766e13a' -> 'a91029b90bf9415a89a2f70736759ad3' move scheduled
I20240501 14:10:17.448298 402 rebalancer_tool.cc:1136] tablet 7628284cb4484ee5b87173790179def0: 'a91029b90bf9415a89a2f70736759ad3' -> 'd35c16dcb506426993811bb2519d7db3' move completed: OK
I20240501 14:10:17.689390 402 rebalancer_tool.cc:1136] tablet 42636717a44e4e3782037c48d8913307: '1d0bb69611af4876a2b3b33c4766e13a' -> '93f2518fbadf4f9991cda93afacb363e' move completed: OK
I20240501 14:11:19.695245 402 rebalancer_tool.cc:1084] tablet 3110aca9b1d643e4b07d0a869015b6f7: 'a91029b90bf9415a89a2f70736759ad3' -> 'db935501e9104de5b15c8780681d6c38' move ignored: Timed out: ChangeConfig RPC to 127.22.252.129:41387 timed out after 60.000s (SENT)
I20240501 14:11:19.695343 402 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 5f8a0cfdfdca4148a75686dc88059cf3 (127.22.252.134:33497): unacceptable health status UNAVAILABLE
I20240501 14:11:19.944764 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 29933
W20240501 14:11:19.954985 32752 connection.cc:535] client connection to 127.22.252.129:41387 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:11:19.955608 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 30064
W20240501 14:11:19.955641 32752 consensus_peers.cc:485] T 90f51696d59c4f11bd44050fde07c276 P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer 93f2518fbadf4f9991cda93afacb363e (127.22.252.129:41387): Couldn't send request to peer 93f2518fbadf4f9991cda93afacb363e. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.129:41387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:11:19.964226 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 30195
I20240501 14:11:19.973186 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 30326
I20240501 14:11:19.981271 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 32341
W20240501 14:11:19.983074 32753 consensus_peers.cc:485] T dbf1233541ad47eda949653a1b486dee P 9ecd907cf5634ac98f3f54f3dbec6849 -> Peer d35c16dcb506426993811bb2519d7db3 (127.22.252.130:36457): Couldn't send request to peer d35c16dcb506426993811bb2519d7db3. Status: Network error: Client connection negotiation failed: client connection to 127.22.252.130:36457: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:11:19.988904 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 32734
I20240501 14:11:19.996707 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 29842
2024-05-01T14:11:19Z chronyd exiting
I20240501 14:11:20.014173 23538 test_util.cc:175] -----------------------------------------------
I20240501 14:11:20.014219 23538 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714572524655928-23538-0
[ FAILED ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0, where GetParam() = 4-byte object <00-00 00-00> (125016 ms)
[----------] 1 test from DDLDuringRebalancingTest (125016 ms total)
[----------] 1 test from RebalancingDuringElectionStormTest
[ RUN ] RebalancingDuringElectionStormTest.RoundRobin/0
I20240501 14:11:20.014808 23538 test_util.cc:268] Using random seed: 260615126
I20240501 14:11:20.015496 23538 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:11:20.015533 23538 ts_itest-base.cc:113] --------------
I20240501 14:11:20.015543 23538 ts_itest-base.cc:114] 8 tablet servers
I20240501 14:11:20.015553 23538 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:11:20.015561 23538 ts_itest-base.cc:116] --------------
2024-05-01T14:11:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:11:20Z Disabled control of system clock
I20240501 14:11:20.023046 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.252.190:43343
--webserver_interface=127.22.252.190
--webserver_port=0
--builtin_ntp_servers=127.22.252.148:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.252.190:43343
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 14:11:20.122757 14713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:11:20.122964 14713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:11:20.123000 14713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:11:20.125948 14713 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:11:20.126015 14713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:11:20.126062 14713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:11:20.126096 14713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:11:20.126125 14713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:11:20.129719 14713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.252.190:43343
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.252.190:43343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.252.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.14713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:11:20.130597 14713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:11:20.131727 14713 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:11:20.139146 14721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:11:20.139735 14718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:11:20.140357 14713 server_base.cc:1034] running on GCE node
W20240501 14:11:20.141177 14719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:11:20.141641 14713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:11:20.142460 14713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:11:20.143599 14713 hybrid_clock.cc:648] HybridClock initialized: now 1714572680143577 us; error 37 us; skew 500 ppm
I20240501 14:11:20.145331 14713 webserver.cc:460] Webserver started at http://127.22.252.190:41013/ using document root <none> and password file <none>
I20240501 14:11:20.145830 14713 fs_manager.cc:359] Metadata directory not provided
I20240501 14:11:20.145900 14713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:11:20.146059 14713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:11:20.147523 14713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e71ec0a1589f480ab9cfeec4a08964fb"
format_stamp: "Formatted at 2024-05-01 14:11:20 on dist-test-slave-bkzb"
I20240501 14:11:20.147998 14713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e71ec0a1589f480ab9cfeec4a08964fb"
format_stamp: "Formatted at 2024-05-01 14:11:20 on dist-test-slave-bkzb"
I20240501 14:11:20.151026 14713 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:11:20.152972 14727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:20.153864 14713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:11:20.153977 14713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e71ec0a1589f480ab9cfeec4a08964fb"
format_stamp: "Formatted at 2024-05-01 14:11:20 on dist-test-slave-bkzb"
I20240501 14:11:20.154048 14713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:11:20.173633 14713 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:11:20.174221 14713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:11:20.174367 14713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:11:20.180822 14713 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.190:43343
I20240501 14:11:20.180830 14779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.190:43343 every 8 connection(s)
I20240501 14:11:20.181707 14713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:11:20.184127 14780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:20.187319 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 14713
I20240501 14:11:20.187433 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:11:20.189347 14780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb: Bootstrap starting.
I20240501 14:11:20.191277 14780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:20.191936 14780 log.cc:824] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb: Log is configured to *not* fsync() on all Append() calls
I20240501 14:11:20.193544 14780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb: No bootstrap required, opened a new log
I20240501 14:11:20.196475 14780 raft_consensus.cc:348] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71ec0a1589f480ab9cfeec4a08964fb" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 43343 } }
I20240501 14:11:20.196655 14780 raft_consensus.cc:374] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:20.196729 14780 raft_consensus.cc:729] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e71ec0a1589f480ab9cfeec4a08964fb, State: Initialized, Role: FOLLOWER
I20240501 14:11:20.197100 14780 consensus_queue.cc:260] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71ec0a1589f480ab9cfeec4a08964fb" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 43343 } }
I20240501 14:11:20.197218 14780 raft_consensus.cc:388] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:11:20.197285 14780 raft_consensus.cc:482] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:11:20.197392 14780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:20.198117 14780 raft_consensus.cc:504] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71ec0a1589f480ab9cfeec4a08964fb" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 43343 } }
I20240501 14:11:20.198415 14780 leader_election.cc:302] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e71ec0a1589f480ab9cfeec4a08964fb; no voters:
I20240501 14:11:20.198666 14780 leader_election.cc:288] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:11:20.198737 14785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:11:20.199309 14785 raft_consensus.cc:686] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb [term 1 LEADER]: Becoming Leader. State: Replica: e71ec0a1589f480ab9cfeec4a08964fb, State: Running, Role: LEADER
I20240501 14:11:20.199561 14785 consensus_queue.cc:237] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71ec0a1589f480ab9cfeec4a08964fb" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 43343 } }
I20240501 14:11:20.199622 14780 sys_catalog.cc:564] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:11:20.200840 14787 sys_catalog.cc:455] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e71ec0a1589f480ab9cfeec4a08964fb. Latest consensus state: current_term: 1 leader_uuid: "e71ec0a1589f480ab9cfeec4a08964fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71ec0a1589f480ab9cfeec4a08964fb" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 43343 } } }
I20240501 14:11:20.200969 14787 sys_catalog.cc:458] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb [sys.catalog]: This master's current role is: LEADER
I20240501 14:11:20.201321 14794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:11:20.201284 14786 sys_catalog.cc:455] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e71ec0a1589f480ab9cfeec4a08964fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71ec0a1589f480ab9cfeec4a08964fb" member_type: VOTER last_known_addr { host: "127.22.252.190" port: 43343 } } }
I20240501 14:11:20.201380 14786 sys_catalog.cc:458] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb [sys.catalog]: This master's current role is: LEADER
I20240501 14:11:20.203665 14794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:11:20.208351 14794 catalog_manager.cc:1347] Generated new cluster ID: 34f148a492654cd38e7b895ca210510a
I20240501 14:11:20.208415 14794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:11:20.216907 14794 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:11:20.217978 14794 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:11:20.227399 14794 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e71ec0a1589f480ab9cfeec4a08964fb: Generated new TSK 0
I20240501 14:11:20.228098 14794 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:11:20.230891 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.129:0
--local_ip_for_outbound_sockets=127.22.252.129
--webserver_interface=127.22.252.129
--webserver_port=0
--tserver_master_addrs=127.22.252.190:43343
--builtin_ntp_servers=127.22.252.148:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:11:20.332168 14804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:11:20.332409 14804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:11:20.332479 14804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:11:20.335466 14804 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:11:20.335541 14804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:11:20.335631 14804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.129
I20240501 14:11:20.339182 14804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.252.129
--webserver_port=0
--tserver_master_addrs=127.22.252.190:43343
--never_fsync=true
--heap_profile_path=/tmp/kudu.14804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.129
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:11:20.339998 14804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:11:20.341053 14804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:11:20.349102 14810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:11:20.349686 14809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:11:20.350688 14812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:11:20.351431 14804 server_base.cc:1034] running on GCE node
I20240501 14:11:20.351759 14804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:11:20.352286 14804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:11:20.353438 14804 hybrid_clock.cc:648] HybridClock initialized: now 1714572680353416 us; error 39 us; skew 500 ppm
I20240501 14:11:20.355214 14804 webserver.cc:460] Webserver started at http://127.22.252.129:36211/ using document root <none> and password file <none>
I20240501 14:11:20.355748 14804 fs_manager.cc:359] Metadata directory not provided
I20240501 14:11:20.355805 14804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:11:20.355966 14804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:11:20.357621 14804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "00035488ce0740898f6f92a508b01a05"
format_stamp: "Formatted at 2024-05-01 14:11:20 on dist-test-slave-bkzb"
I20240501 14:11:20.358147 14804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "00035488ce0740898f6f92a508b01a05"
format_stamp: "Formatted at 2024-05-01 14:11:20 on dist-test-slave-bkzb"
I20240501 14:11:20.361155 14804 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:11:20.363121 14818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:20.363981 14804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:11:20.364076 14804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "00035488ce0740898f6f92a508b01a05"
format_stamp: "Formatted at 2024-05-01 14:11:20 on dist-test-slave-bkzb"
I20240501 14:11:20.364143 14804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:11:20.386108 14804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:11:20.386564 14804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:11:20.386695 14804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:11:20.387128 14804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:11:20.387876 14804 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:11:20.387921 14804 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:20.387985 14804 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:11:20.388010 14804 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:20.395550 14804 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.129:33411
I20240501 14:11:20.395579 14931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.129:33411 every 8 connection(s)
I20240501 14:11:20.396389 14804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:11:20.404415 14932 heartbeater.cc:347] Connected to a master server at 127.22.252.190:43343
I20240501 14:11:20.404592 14932 heartbeater.cc:464] Registering TS with master...
I20240501 14:11:20.404974 14932 heartbeater.cc:511] Master 127.22.252.190:43343 requested a full tablet report, sending...
I20240501 14:11:20.406164 14744 ts_manager.cc:194] Registered new tserver with Master: 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411)
I20240501 14:11:20.406440 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 14804
I20240501 14:11:20.406538 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:11:20.407322 14744 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.129:54789
I20240501 14:11:20.408613 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.130:0
--local_ip_for_outbound_sockets=127.22.252.130
--webserver_interface=127.22.252.130
--webserver_port=0
--tserver_master_addrs=127.22.252.190:43343
--builtin_ntp_servers=127.22.252.148:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:11:20.507277 14935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:11:20.507486 14935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:11:20.507551 14935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:11:20.510470 14935 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:11:20.510531 14935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:11:20.510622 14935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.130
I20240501 14:11:20.513939 14935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.252.130
--webserver_port=0
--tserver_master_addrs=127.22.252.190:43343
--never_fsync=true
--heap_profile_path=/tmp/kudu.14935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.130
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:11:20.514762 14935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:11:20.515777 14935 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:11:20.523597 14943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:11:20.524111 14941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:11:20.524919 14940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:11:20.525588 14935 server_base.cc:1034] running on GCE node
I20240501 14:11:20.525956 14935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:11:20.526428 14935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:11:20.527580 14935 hybrid_clock.cc:648] HybridClock initialized: now 1714572680527565 us; error 27 us; skew 500 ppm
I20240501 14:11:20.529306 14935 webserver.cc:460] Webserver started at http://127.22.252.130:46495/ using document root <none> and password file <none>
I20240501 14:11:20.529814 14935 fs_manager.cc:359] Metadata directory not provided
I20240501 14:11:20.529872 14935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:11:20.530035 14935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:11:20.531543 14935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "5f49c2edf1b54d83811340aaea48e682"
format_stamp: "Formatted at 2024-05-01 14:11:20 on dist-test-slave-bkzb"
I20240501 14:11:20.531978 14935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "5f49c2edf1b54d83811340aaea48e682"
format_stamp: "Formatted at 2024-05-01 14:11:20 on dist-test-slave-bkzb"
I20240501 14:11:20.535034 14935 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20240501 14:11:20.537031 14949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:20.537863 14935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:11:20.537966 14935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "5f49c2edf1b54d83811340aaea48e682"
format_stamp: "Formatted at 2024-05-01 14:11:20 on dist-test-slave-bkzb"
I20240501 14:11:20.538069 14935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:11:20.556830 14935 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:11:20.557300 14935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:11:20.557441 14935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:11:20.557859 14935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:11:20.558682 14935 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:11:20.558728 14935 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:20.558786 14935 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:11:20.558815 14935 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:20.566001 14935 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.130:40469
I20240501 14:11:20.566025 15062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.130:40469 every 8 connection(s)
I20240501 14:11:20.566821 14935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:11:20.573923 15063 heartbeater.cc:347] Connected to a master server at 127.22.252.190:43343
I20240501 14:11:20.573953 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 14935
I20240501 14:11:20.574057 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:11:20.574101 15063 heartbeater.cc:464] Registering TS with master...
I20240501 14:11:20.574445 15063 heartbeater.cc:511] Master 127.22.252.190:43343 requested a full tablet report, sending...
I20240501 14:11:20.575246 14744 ts_manager.cc:194] Registered new tserver with Master: 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130:40469)
I20240501 14:11:20.575862 14744 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.130:38307
I20240501 14:11:20.575978 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.131:0
--local_ip_for_outbound_sockets=127.22.252.131
--webserver_interface=127.22.252.131
--webserver_port=0
--tserver_master_addrs=127.22.252.190:43343
--builtin_ntp_servers=127.22.252.148:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:11:20.674078 15066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:11:20.674273 15066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:11:20.674335 15066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:11:20.677148 15066 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:11:20.677209 15066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:11:20.677295 15066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.131
I20240501 14:11:20.680645 15066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.252.131
--webserver_port=0
--tserver_master_addrs=127.22.252.190:43343
--never_fsync=true
--heap_profile_path=/tmp/kudu.15066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.131
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:11:20.681357 15066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:11:20.682308 15066 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:11:20.690026 15071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:11:20.690789 15072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:11:20.691329 15074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:11:20.691987 15066 server_base.cc:1034] running on GCE node
I20240501 14:11:20.692451 15066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:11:20.692976 15066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:11:20.694126 15066 hybrid_clock.cc:648] HybridClock initialized: now 1714572680694095 us; error 43 us; skew 500 ppm
I20240501 14:11:20.695911 15066 webserver.cc:460] Webserver started at http://127.22.252.131:38287/ using document root <none> and password file <none>
I20240501 14:11:20.696416 15066 fs_manager.cc:359] Metadata directory not provided
I20240501 14:11:20.696468 15066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:11:20.696666 15066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:11:20.698227 15066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "dcc6733e9b3741578b9bc25604dd3bc6"
format_stamp: "Formatted at 2024-05-01 14:11:20 on dist-test-slave-bkzb"
I20240501 14:11:20.698683 15066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "dcc6733e9b3741578b9bc25604dd3bc6"
format_stamp: "Formatted at 2024-05-01 14:11:20 on dist-test-slave-bkzb"
I20240501 14:11:20.701956 15066 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:11:20.703794 15080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:20.704736 15066 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:11:20.704865 15066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "dcc6733e9b3741578b9bc25604dd3bc6"
format_stamp: "Formatted at 2024-05-01 14:11:20 on dist-test-slave-bkzb"
I20240501 14:11:20.704952 15066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:11:20.746871 15066 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:11:20.747478 15066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:11:20.747629 15066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:11:20.748144 15066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:11:20.749007 15066 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:11:20.749054 15066 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:20.749114 15066 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:11:20.749145 15066 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:20.757090 15066 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.131:39965
I20240501 14:11:20.757117 15193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.131:39965 every 8 connection(s)
I20240501 14:11:20.758023 15066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:11:20.761293 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 15066
I20240501 14:11:20.761401 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:11:20.763365 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.132:0
--local_ip_for_outbound_sockets=127.22.252.132
--webserver_interface=127.22.252.132
--webserver_port=0
--tserver_master_addrs=127.22.252.190:43343
--builtin_ntp_servers=127.22.252.148:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:11:20.766507 15194 heartbeater.cc:347] Connected to a master server at 127.22.252.190:43343
I20240501 14:11:20.766731 15194 heartbeater.cc:464] Registering TS with master...
I20240501 14:11:20.767308 15194 heartbeater.cc:511] Master 127.22.252.190:43343 requested a full tablet report, sending...
I20240501 14:11:20.768337 14744 ts_manager.cc:194] Registered new tserver with Master: dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131:39965)
I20240501 14:11:20.768886 14744 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.131:55719
W20240501 14:11:20.864949 15197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:11:20.865126 15197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:11:20.865176 15197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:11:20.868101 15197 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:11:20.868151 15197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:11:20.868257 15197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.132
I20240501 14:11:20.871546 15197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.252.132
--webserver_port=0
--tserver_master_addrs=127.22.252.190:43343
--never_fsync=true
--heap_profile_path=/tmp/kudu.15197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.132
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:11:20.872313 15197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:11:20.873333 15197 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:11:20.880985 15205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:11:20.881493 15202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:11:20.882020 15203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:11:20.882742 15197 server_base.cc:1034] running on GCE node
I20240501 14:11:20.883065 15197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:11:20.883514 15197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:11:20.884650 15197 hybrid_clock.cc:648] HybridClock initialized: now 1714572680884640 us; error 24 us; skew 500 ppm
I20240501 14:11:20.886286 15197 webserver.cc:460] Webserver started at http://127.22.252.132:40489/ using document root <none> and password file <none>
I20240501 14:11:20.886765 15197 fs_manager.cc:359] Metadata directory not provided
I20240501 14:11:20.886826 15197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:11:20.886982 15197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:11:20.888543 15197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948"
format_stamp: "Formatted at 2024-05-01 14:11:20 on dist-test-slave-bkzb"
I20240501 14:11:20.889029 15197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948"
format_stamp: "Formatted at 2024-05-01 14:11:20 on dist-test-slave-bkzb"
I20240501 14:11:20.891958 15197 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:11:20.893918 15211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:20.894801 15197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:11:20.894901 15197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948"
format_stamp: "Formatted at 2024-05-01 14:11:20 on dist-test-slave-bkzb"
I20240501 14:11:20.894979 15197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:11:20.906459 15197 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:11:20.906903 15197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:11:20.907042 15197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:11:20.907486 15197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:11:20.908319 15197 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:11:20.908366 15197 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:20.908419 15197 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:11:20.908445 15197 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:20.916046 15197 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.132:40939
I20240501 14:11:20.916070 15324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.132:40939 every 8 connection(s)
I20240501 14:11:20.916889 15197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:11:20.919131 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 15197
I20240501 14:11:20.919260 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:11:20.921723 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.133:0
--local_ip_for_outbound_sockets=127.22.252.133
--webserver_interface=127.22.252.133
--webserver_port=0
--tserver_master_addrs=127.22.252.190:43343
--builtin_ntp_servers=127.22.252.148:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:11:20.925550 15325 heartbeater.cc:347] Connected to a master server at 127.22.252.190:43343
I20240501 14:11:20.925765 15325 heartbeater.cc:464] Registering TS with master...
I20240501 14:11:20.926154 15325 heartbeater.cc:511] Master 127.22.252.190:43343 requested a full tablet report, sending...
I20240501 14:11:20.927011 14744 ts_manager.cc:194] Registered new tserver with Master: 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132:40939)
I20240501 14:11:20.927806 14744 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.132:51023
W20240501 14:11:21.020690 15328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:11:21.020866 15328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:11:21.020905 15328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:11:21.023531 15328 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:11:21.023578 15328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:11:21.023634 15328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.133
I20240501 14:11:21.026965 15328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.252.133
--webserver_port=0
--tserver_master_addrs=127.22.252.190:43343
--never_fsync=true
--heap_profile_path=/tmp/kudu.15328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.133
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:11:21.027648 15328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:11:21.028632 15328 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:11:21.036532 15334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:11:21.036960 15336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:11:21.037761 15333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:11:21.038473 15328 server_base.cc:1034] running on GCE node
I20240501 14:11:21.038820 15328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:11:21.039300 15328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:11:21.040445 15328 hybrid_clock.cc:648] HybridClock initialized: now 1714572681040430 us; error 25 us; skew 500 ppm
I20240501 14:11:21.042090 15328 webserver.cc:460] Webserver started at http://127.22.252.133:38819/ using document root <none> and password file <none>
I20240501 14:11:21.042549 15328 fs_manager.cc:359] Metadata directory not provided
I20240501 14:11:21.042608 15328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:11:21.042783 15328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:11:21.044317 15328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "22ecb2ee7dea4346a7a5662bb8c80c03"
format_stamp: "Formatted at 2024-05-01 14:11:21 on dist-test-slave-bkzb"
I20240501 14:11:21.044735 15328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "22ecb2ee7dea4346a7a5662bb8c80c03"
format_stamp: "Formatted at 2024-05-01 14:11:21 on dist-test-slave-bkzb"
I20240501 14:11:21.047605 15328 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20240501 14:11:21.049587 15342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:21.050463 15328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:11:21.050566 15328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "22ecb2ee7dea4346a7a5662bb8c80c03"
format_stamp: "Formatted at 2024-05-01 14:11:21 on dist-test-slave-bkzb"
I20240501 14:11:21.050648 15328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:11:21.060870 15328 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:11:21.061301 15328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:11:21.061432 15328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:11:21.061857 15328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:11:21.062623 15328 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:11:21.062669 15328 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:21.062721 15328 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:11:21.062754 15328 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:21.070662 15328 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.133:38171
I20240501 14:11:21.070689 15455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.133:38171 every 8 connection(s)
I20240501 14:11:21.071393 15328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:11:21.077006 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 15328
I20240501 14:11:21.077126 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:11:21.078904 15456 heartbeater.cc:347] Connected to a master server at 127.22.252.190:43343
I20240501 14:11:21.079092 15456 heartbeater.cc:464] Registering TS with master...
I20240501 14:11:21.079217 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.134:0
--local_ip_for_outbound_sockets=127.22.252.134
--webserver_interface=127.22.252.134
--webserver_port=0
--tserver_master_addrs=127.22.252.190:43343
--builtin_ntp_servers=127.22.252.148:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:11:21.079502 15456 heartbeater.cc:511] Master 127.22.252.190:43343 requested a full tablet report, sending...
I20240501 14:11:21.080356 14743 ts_manager.cc:194] Registered new tserver with Master: 22ecb2ee7dea4346a7a5662bb8c80c03 (127.22.252.133:38171)
I20240501 14:11:21.081079 14743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.133:45505
W20240501 14:11:21.177989 15459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:11:21.178175 15459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:11:21.178217 15459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:11:21.181142 15459 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:11:21.181208 15459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:11:21.181294 15459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.134
I20240501 14:11:21.185801 15459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.252.134
--webserver_port=0
--tserver_master_addrs=127.22.252.190:43343
--never_fsync=true
--heap_profile_path=/tmp/kudu.15459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.134
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:11:21.186611 15459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:11:21.187724 15459 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:11:21.196295 15467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:11:21.196830 15464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:11:21.197364 15465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:11:21.198282 15459 server_base.cc:1034] running on GCE node
I20240501 14:11:21.198613 15459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:11:21.199057 15459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:11:21.200192 15459 hybrid_clock.cc:648] HybridClock initialized: now 1714572681200175 us; error 31 us; skew 500 ppm
I20240501 14:11:21.201890 15459 webserver.cc:460] Webserver started at http://127.22.252.134:38349/ using document root <none> and password file <none>
I20240501 14:11:21.202317 15459 fs_manager.cc:359] Metadata directory not provided
I20240501 14:11:21.202363 15459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:11:21.202507 15459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:11:21.203889 15459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "e991d79457f64d7a82d7d9ae4f20e0bd"
format_stamp: "Formatted at 2024-05-01 14:11:21 on dist-test-slave-bkzb"
I20240501 14:11:21.204340 15459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "e991d79457f64d7a82d7d9ae4f20e0bd"
format_stamp: "Formatted at 2024-05-01 14:11:21 on dist-test-slave-bkzb"
I20240501 14:11:21.207458 15459 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20240501 14:11:21.209419 15473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:21.210320 15459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:11:21.210419 15459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e991d79457f64d7a82d7d9ae4f20e0bd"
format_stamp: "Formatted at 2024-05-01 14:11:21 on dist-test-slave-bkzb"
I20240501 14:11:21.210494 15459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:11:21.230494 15459 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:11:21.230986 15459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:11:21.231132 15459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:11:21.231570 15459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:11:21.232383 15459 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:11:21.232430 15459 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:21.232484 15459 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:11:21.232513 15459 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:21.240581 15459 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.134:35651
I20240501 14:11:21.240636 15586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.134:35651 every 8 connection(s)
I20240501 14:11:21.241503 15459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:11:21.245033 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 15459
I20240501 14:11:21.245157 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:11:21.247301 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.135:0
--local_ip_for_outbound_sockets=127.22.252.135
--webserver_interface=127.22.252.135
--webserver_port=0
--tserver_master_addrs=127.22.252.190:43343
--builtin_ntp_servers=127.22.252.148:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:11:21.250367 15587 heartbeater.cc:347] Connected to a master server at 127.22.252.190:43343
I20240501 14:11:21.250560 15587 heartbeater.cc:464] Registering TS with master...
I20240501 14:11:21.250993 15587 heartbeater.cc:511] Master 127.22.252.190:43343 requested a full tablet report, sending...
I20240501 14:11:21.251911 14743 ts_manager.cc:194] Registered new tserver with Master: e991d79457f64d7a82d7d9ae4f20e0bd (127.22.252.134:35651)
I20240501 14:11:21.252465 14743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.134:58777
W20240501 14:11:21.347715 15590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:11:21.347934 15590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:11:21.348006 15590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:11:21.350904 15590 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:11:21.350971 15590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:11:21.351060 15590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.135
I20240501 14:11:21.354458 15590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.252.135
--webserver_port=0
--tserver_master_addrs=127.22.252.190:43343
--never_fsync=true
--heap_profile_path=/tmp/kudu.15590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.135
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:11:21.355262 15590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:11:21.356315 15590 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:11:21.364007 15595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:11:21.364686 15598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:11:21.365247 15596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:11:21.365886 15590 server_base.cc:1034] running on GCE node
I20240501 14:11:21.366189 15590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:11:21.366663 15590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:11:21.367801 15590 hybrid_clock.cc:648] HybridClock initialized: now 1714572681367781 us; error 36 us; skew 500 ppm
I20240501 14:11:21.369428 15590 webserver.cc:460] Webserver started at http://127.22.252.135:41763/ using document root <none> and password file <none>
I20240501 14:11:21.369956 15590 fs_manager.cc:359] Metadata directory not provided
I20240501 14:11:21.370012 15590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:11:21.370167 15590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:11:21.371606 15590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "f5c529eab5664b33a027ec73c8b7acea"
format_stamp: "Formatted at 2024-05-01 14:11:21 on dist-test-slave-bkzb"
I20240501 14:11:21.372036 15590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "f5c529eab5664b33a027ec73c8b7acea"
format_stamp: "Formatted at 2024-05-01 14:11:21 on dist-test-slave-bkzb"
I20240501 14:11:21.374959 15590 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:11:21.376772 15604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:21.377599 15590 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:11:21.377693 15590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "f5c529eab5664b33a027ec73c8b7acea"
format_stamp: "Formatted at 2024-05-01 14:11:21 on dist-test-slave-bkzb"
I20240501 14:11:21.377760 15590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:11:21.393126 15590 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:11:21.393594 15590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:11:21.393743 15590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:11:21.394150 15590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:11:21.394850 15590 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:11:21.394894 15590 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:21.394943 15590 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:11:21.394970 15590 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:21.403299 15590 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.135:38183
I20240501 14:11:21.403343 15717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.135:38183 every 8 connection(s)
I20240501 14:11:21.404860 15590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:11:21.409265 14932 heartbeater.cc:503] Master 127.22.252.190:43343 was elected leader, sending a full tablet report...
I20240501 14:11:21.412993 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 15590
I20240501 14:11:21.413117 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 14:11:21.414376 15718 heartbeater.cc:347] Connected to a master server at 127.22.252.190:43343
I20240501 14:11:21.414568 15718 heartbeater.cc:464] Registering TS with master...
I20240501 14:11:21.415059 15718 heartbeater.cc:511] Master 127.22.252.190:43343 requested a full tablet report, sending...
I20240501 14:11:21.415145 23538 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
/tmp/dist-test-taskIGCbFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.252.136:0
--local_ip_for_outbound_sockets=127.22.252.136
--webserver_interface=127.22.252.136
--webserver_port=0
--tserver_master_addrs=127.22.252.190:43343
--builtin_ntp_servers=127.22.252.148:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:11:21.415853 14744 ts_manager.cc:194] Registered new tserver with Master: f5c529eab5664b33a027ec73c8b7acea (127.22.252.135:38183)
I20240501 14:11:21.416452 14744 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.135:47519
W20240501 14:11:21.515770 15721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:11:21.515960 15721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:11:21.516026 15721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:11:21.518723 15721 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:11:21.518788 15721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:11:21.518880 15721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.252.136
I20240501 14:11:21.522267 15721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.252.148:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.252.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.252.136
--webserver_port=0
--tserver_master_addrs=127.22.252.190:43343
--never_fsync=true
--heap_profile_path=/tmp/kudu.15721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.252.136
--log_dir=/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:11:21.523033 15721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:11:21.524022 15721 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:11:21.531939 15729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:11:21.532452 15727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:11:21.533018 15726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:11:21.533864 15721 server_base.cc:1034] running on GCE node
I20240501 14:11:21.534241 15721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:11:21.534715 15721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:11:21.535864 15721 hybrid_clock.cc:648] HybridClock initialized: now 1714572681535856 us; error 27 us; skew 500 ppm
I20240501 14:11:21.537708 15721 webserver.cc:460] Webserver started at http://127.22.252.136:36019/ using document root <none> and password file <none>
I20240501 14:11:21.538234 15721 fs_manager.cc:359] Metadata directory not provided
I20240501 14:11:21.538292 15721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:11:21.538451 15721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:11:21.540012 15721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "a86210b8a2aa4beeb875098afa37d98c"
format_stamp: "Formatted at 2024-05-01 14:11:21 on dist-test-slave-bkzb"
I20240501 14:11:21.540469 15721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "a86210b8a2aa4beeb875098afa37d98c"
format_stamp: "Formatted at 2024-05-01 14:11:21 on dist-test-slave-bkzb"
I20240501 14:11:21.543623 15721 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20240501 14:11:21.545636 15735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:21.546502 15721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240501 14:11:21.546597 15721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "a86210b8a2aa4beeb875098afa37d98c"
format_stamp: "Formatted at 2024-05-01 14:11:21 on dist-test-slave-bkzb"
I20240501 14:11:21.546665 15721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:11:21.567816 15721 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:11:21.568322 15721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:11:21.568452 15721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:11:21.568907 15721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:11:21.569665 15721 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:11:21.569725 15721 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:21.569778 15721 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:11:21.569803 15721 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:11:21.577015 15721 rpc_server.cc:307] RPC server started. Bound to: 127.22.252.136:41829
I20240501 14:11:21.577050 15848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.252.136:41829 every 8 connection(s)
I20240501 14:11:21.577666 15063 heartbeater.cc:503] Master 127.22.252.190:43343 was elected leader, sending a full tablet report...
I20240501 14:11:21.577790 15721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:11:21.580547 23538 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu as pid 15721
I20240501 14:11:21.580672 23538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIGCbFd/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714572524655928-23538-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 14:11:21.585709 15849 heartbeater.cc:347] Connected to a master server at 127.22.252.190:43343
I20240501 14:11:21.585882 15849 heartbeater.cc:464] Registering TS with master...
I20240501 14:11:21.586237 15849 heartbeater.cc:511] Master 127.22.252.190:43343 requested a full tablet report, sending...
I20240501 14:11:21.586977 14744 ts_manager.cc:194] Registered new tserver with Master: a86210b8a2aa4beeb875098afa37d98c (127.22.252.136:41829)
I20240501 14:11:21.587433 14744 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.252.136:51441
I20240501 14:11:21.594313 23538 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:11:21.603188 23538 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:11:21.603246 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 15328
I20240501 14:11:21.608772 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 15459
I20240501 14:11:21.614104 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 15590
I20240501 14:11:21.619282 23538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIGCbFd/build/debug/bin/kudu with pid 15721
I20240501 14:11:21.771185 15194 heartbeater.cc:503] Master 127.22.252.190:43343 was elected leader, sending a full tablet report...
I20240501 14:11:21.929545 15325 heartbeater.cc:503] Master 127.22.252.190:43343 was elected leader, sending a full tablet report...
I20240501 14:11:25.375779 14744 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41026:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:11:25.395393 15258 tablet_service.cc:1467] Processing CreateTablet for tablet e0d2be3d67f64fdc984578202c77ff12 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2beb3aecffa947ba9edbe8d9d6d51e1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:25.395393 14865 tablet_service.cc:1467] Processing CreateTablet for tablet 3c78677faa8a4ad28d7209b0360576bc (DEFAULT_TABLE table=rebalance_test_table_0 [id=2beb3aecffa947ba9edbe8d9d6d51e1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:25.395460 14997 tablet_service.cc:1467] Processing CreateTablet for tablet be9fc0ba767947779aab78c6e279d25a (DEFAULT_TABLE table=rebalance_test_table_0 [id=2beb3aecffa947ba9edbe8d9d6d51e1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:25.395639 14996 tablet_service.cc:1467] Processing CreateTablet for tablet 3c78677faa8a4ad28d7209b0360576bc (DEFAULT_TABLE table=rebalance_test_table_0 [id=2beb3aecffa947ba9edbe8d9d6d51e1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:25.395395 14866 tablet_service.cc:1467] Processing CreateTablet for tablet be9fc0ba767947779aab78c6e279d25a (DEFAULT_TABLE table=rebalance_test_table_0 [id=2beb3aecffa947ba9edbe8d9d6d51e1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:25.396375 14865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c78677faa8a4ad28d7209b0360576bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:25.396376 15258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d2be3d67f64fdc984578202c77ff12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:25.396379 14997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be9fc0ba767947779aab78c6e279d25a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:25.396700 14866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be9fc0ba767947779aab78c6e279d25a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:25.397547 14996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c78677faa8a4ad28d7209b0360576bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:25.398061 14864 tablet_service.cc:1467] Processing CreateTablet for tablet e0d2be3d67f64fdc984578202c77ff12 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2beb3aecffa947ba9edbe8d9d6d51e1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:25.398278 14864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d2be3d67f64fdc984578202c77ff12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:25.402765 15128 tablet_service.cc:1467] Processing CreateTablet for tablet 3c78677faa8a4ad28d7209b0360576bc (DEFAULT_TABLE table=rebalance_test_table_0 [id=2beb3aecffa947ba9edbe8d9d6d51e1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:25.403604 15128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c78677faa8a4ad28d7209b0360576bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:25.403998 15127 tablet_service.cc:1467] Processing CreateTablet for tablet e0d2be3d67f64fdc984578202c77ff12 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2beb3aecffa947ba9edbe8d9d6d51e1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:25.404220 15127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d2be3d67f64fdc984578202c77ff12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:25.404660 15259 tablet_service.cc:1467] Processing CreateTablet for tablet be9fc0ba767947779aab78c6e279d25a (DEFAULT_TABLE table=rebalance_test_table_0 [id=2beb3aecffa947ba9edbe8d9d6d51e1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:25.404930 15259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be9fc0ba767947779aab78c6e279d25a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:25.405872 15870 tablet_bootstrap.cc:492] T be9fc0ba767947779aab78c6e279d25a P 5f49c2edf1b54d83811340aaea48e682: Bootstrap starting.
I20240501 14:11:25.407492 15870 tablet_bootstrap.cc:654] T be9fc0ba767947779aab78c6e279d25a P 5f49c2edf1b54d83811340aaea48e682: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:25.408155 15870 log.cc:824] T be9fc0ba767947779aab78c6e279d25a P 5f49c2edf1b54d83811340aaea48e682: Log is configured to *not* fsync() on all Append() calls
I20240501 14:11:25.408943 15871 tablet_bootstrap.cc:492] T be9fc0ba767947779aab78c6e279d25a P 00035488ce0740898f6f92a508b01a05: Bootstrap starting.
I20240501 14:11:25.411204 15870 tablet_bootstrap.cc:492] T be9fc0ba767947779aab78c6e279d25a P 5f49c2edf1b54d83811340aaea48e682: No bootstrap required, opened a new log
I20240501 14:11:25.411559 15870 ts_tablet_manager.cc:1367] T be9fc0ba767947779aab78c6e279d25a P 5f49c2edf1b54d83811340aaea48e682: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:11:25.413192 15871 tablet_bootstrap.cc:654] T be9fc0ba767947779aab78c6e279d25a P 00035488ce0740898f6f92a508b01a05: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:25.413915 15873 tablet_bootstrap.cc:492] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6: Bootstrap starting.
I20240501 14:11:25.413923 15869 tablet_bootstrap.cc:492] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Bootstrap starting.
I20240501 14:11:25.413973 15871 log.cc:824] T be9fc0ba767947779aab78c6e279d25a P 00035488ce0740898f6f92a508b01a05: Log is configured to *not* fsync() on all Append() calls
I20240501 14:11:25.415845 15869 tablet_bootstrap.cc:654] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:25.416592 15869 log.cc:824] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Log is configured to *not* fsync() on all Append() calls
I20240501 14:11:25.416762 15870 raft_consensus.cc:348] T be9fc0ba767947779aab78c6e279d25a P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.416958 15870 raft_consensus.cc:374] T be9fc0ba767947779aab78c6e279d25a P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:25.417016 15870 raft_consensus.cc:729] T be9fc0ba767947779aab78c6e279d25a P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f49c2edf1b54d83811340aaea48e682, State: Initialized, Role: FOLLOWER
I20240501 14:11:25.417084 15871 tablet_bootstrap.cc:492] T be9fc0ba767947779aab78c6e279d25a P 00035488ce0740898f6f92a508b01a05: No bootstrap required, opened a new log
I20240501 14:11:25.417224 15871 ts_tablet_manager.cc:1367] T be9fc0ba767947779aab78c6e279d25a P 00035488ce0740898f6f92a508b01a05: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240501 14:11:25.417233 15873 tablet_bootstrap.cc:654] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:25.417466 15870 consensus_queue.cc:260] T be9fc0ba767947779aab78c6e279d25a P 5f49c2edf1b54d83811340aaea48e682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.417883 15873 log.cc:824] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6: Log is configured to *not* fsync() on all Append() calls
I20240501 14:11:25.418170 15869 tablet_bootstrap.cc:492] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948: No bootstrap required, opened a new log
I20240501 14:11:25.418295 15870 ts_tablet_manager.cc:1398] T be9fc0ba767947779aab78c6e279d25a P 5f49c2edf1b54d83811340aaea48e682: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240501 14:11:25.418321 15869 ts_tablet_manager.cc:1367] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:11:25.419409 15873 tablet_bootstrap.cc:492] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6: No bootstrap required, opened a new log
I20240501 14:11:25.419545 15873 ts_tablet_manager.cc:1367] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 14:11:25.421140 15871 raft_consensus.cc:348] T be9fc0ba767947779aab78c6e279d25a P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.421416 15871 raft_consensus.cc:374] T be9fc0ba767947779aab78c6e279d25a P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:25.421509 15871 raft_consensus.cc:729] T be9fc0ba767947779aab78c6e279d25a P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00035488ce0740898f6f92a508b01a05, State: Initialized, Role: FOLLOWER
I20240501 14:11:25.421978 15870 tablet_bootstrap.cc:492] T 3c78677faa8a4ad28d7209b0360576bc P 5f49c2edf1b54d83811340aaea48e682: Bootstrap starting.
I20240501 14:11:25.421993 15871 consensus_queue.cc:260] T be9fc0ba767947779aab78c6e279d25a P 00035488ce0740898f6f92a508b01a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.421995 15869 raft_consensus.cc:348] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:25.422170 15869 raft_consensus.cc:374] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:25.422230 15869 raft_consensus.cc:729] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f24c1c43aa2456eb67b2cbb9ed1a948, State: Initialized, Role: FOLLOWER
I20240501 14:11:25.422634 15869 consensus_queue.cc:260] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:25.422859 15870 tablet_bootstrap.cc:654] T 3c78677faa8a4ad28d7209b0360576bc P 5f49c2edf1b54d83811340aaea48e682: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:25.423110 15871 ts_tablet_manager.cc:1398] T be9fc0ba767947779aab78c6e279d25a P 00035488ce0740898f6f92a508b01a05: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20240501 14:11:25.423015 15873 raft_consensus.cc:348] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:25.423173 15873 raft_consensus.cc:374] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:25.423229 15873 raft_consensus.cc:729] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc6733e9b3741578b9bc25604dd3bc6, State: Initialized, Role: FOLLOWER
I20240501 14:11:25.423381 15869 ts_tablet_manager.cc:1398] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 14:11:25.423607 15871 tablet_bootstrap.cc:492] T e0d2be3d67f64fdc984578202c77ff12 P 00035488ce0740898f6f92a508b01a05: Bootstrap starting.
I20240501 14:11:25.423702 15869 tablet_bootstrap.cc:492] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948: Bootstrap starting.
I20240501 14:11:25.424521 15871 tablet_bootstrap.cc:654] T e0d2be3d67f64fdc984578202c77ff12 P 00035488ce0740898f6f92a508b01a05: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:25.424671 15869 tablet_bootstrap.cc:654] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:25.424923 15873 consensus_queue.cc:260] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:25.425676 15871 tablet_bootstrap.cc:492] T e0d2be3d67f64fdc984578202c77ff12 P 00035488ce0740898f6f92a508b01a05: No bootstrap required, opened a new log
I20240501 14:11:25.425705 15873 ts_tablet_manager.cc:1398] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 14:11:25.425792 15871 ts_tablet_manager.cc:1367] T e0d2be3d67f64fdc984578202c77ff12 P 00035488ce0740898f6f92a508b01a05: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:11:25.426683 15870 tablet_bootstrap.cc:492] T 3c78677faa8a4ad28d7209b0360576bc P 5f49c2edf1b54d83811340aaea48e682: No bootstrap required, opened a new log
I20240501 14:11:25.427186 15870 ts_tablet_manager.cc:1367] T 3c78677faa8a4ad28d7209b0360576bc P 5f49c2edf1b54d83811340aaea48e682: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:11:25.427332 15871 raft_consensus.cc:348] T e0d2be3d67f64fdc984578202c77ff12 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:25.427467 15871 raft_consensus.cc:374] T e0d2be3d67f64fdc984578202c77ff12 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:25.427498 15871 raft_consensus.cc:729] T e0d2be3d67f64fdc984578202c77ff12 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00035488ce0740898f6f92a508b01a05, State: Initialized, Role: FOLLOWER
I20240501 14:11:25.427652 15871 consensus_queue.cc:260] T e0d2be3d67f64fdc984578202c77ff12 P 00035488ce0740898f6f92a508b01a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:25.428011 15871 ts_tablet_manager.cc:1398] T e0d2be3d67f64fdc984578202c77ff12 P 00035488ce0740898f6f92a508b01a05: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:11:25.428218 15871 tablet_bootstrap.cc:492] T 3c78677faa8a4ad28d7209b0360576bc P 00035488ce0740898f6f92a508b01a05: Bootstrap starting.
I20240501 14:11:25.428174 15870 raft_consensus.cc:348] T 3c78677faa8a4ad28d7209b0360576bc P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.428345 15870 raft_consensus.cc:374] T 3c78677faa8a4ad28d7209b0360576bc P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:25.428476 15873 tablet_bootstrap.cc:492] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6: Bootstrap starting.
I20240501 14:11:25.428408 15870 raft_consensus.cc:729] T 3c78677faa8a4ad28d7209b0360576bc P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f49c2edf1b54d83811340aaea48e682, State: Initialized, Role: FOLLOWER
I20240501 14:11:25.429234 15871 tablet_bootstrap.cc:654] T 3c78677faa8a4ad28d7209b0360576bc P 00035488ce0740898f6f92a508b01a05: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:25.429234 15870 consensus_queue.cc:260] T 3c78677faa8a4ad28d7209b0360576bc P 5f49c2edf1b54d83811340aaea48e682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.429386 15873 tablet_bootstrap.cc:654] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:25.429620 15870 ts_tablet_manager.cc:1398] T 3c78677faa8a4ad28d7209b0360576bc P 5f49c2edf1b54d83811340aaea48e682: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:11:25.430794 15869 tablet_bootstrap.cc:492] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948: No bootstrap required, opened a new log
I20240501 14:11:25.430886 15869 ts_tablet_manager.cc:1367] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 14:11:25.430999 15873 tablet_bootstrap.cc:492] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6: No bootstrap required, opened a new log
I20240501 14:11:25.431072 15873 ts_tablet_manager.cc:1367] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:11:25.431376 15869 raft_consensus.cc:348] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.431483 15869 raft_consensus.cc:374] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:25.431529 15869 raft_consensus.cc:729] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f24c1c43aa2456eb67b2cbb9ed1a948, State: Initialized, Role: FOLLOWER
I20240501 14:11:25.431664 15869 consensus_queue.cc:260] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.431602 15873 raft_consensus.cc:348] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.431756 15873 raft_consensus.cc:374] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:25.431807 15873 raft_consensus.cc:729] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc6733e9b3741578b9bc25604dd3bc6, State: Initialized, Role: FOLLOWER
I20240501 14:11:25.431941 15869 ts_tablet_manager.cc:1398] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:11:25.431964 15873 consensus_queue.cc:260] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.432322 15873 ts_tablet_manager.cc:1398] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:25.432744 15871 tablet_bootstrap.cc:492] T 3c78677faa8a4ad28d7209b0360576bc P 00035488ce0740898f6f92a508b01a05: No bootstrap required, opened a new log
I20240501 14:11:25.432822 15871 ts_tablet_manager.cc:1367] T 3c78677faa8a4ad28d7209b0360576bc P 00035488ce0740898f6f92a508b01a05: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:11:25.433213 15871 raft_consensus.cc:348] T 3c78677faa8a4ad28d7209b0360576bc P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.433313 15871 raft_consensus.cc:374] T 3c78677faa8a4ad28d7209b0360576bc P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:25.433351 15871 raft_consensus.cc:729] T 3c78677faa8a4ad28d7209b0360576bc P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00035488ce0740898f6f92a508b01a05, State: Initialized, Role: FOLLOWER
I20240501 14:11:25.433475 15871 consensus_queue.cc:260] T 3c78677faa8a4ad28d7209b0360576bc P 00035488ce0740898f6f92a508b01a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.433988 15871 ts_tablet_manager.cc:1398] T 3c78677faa8a4ad28d7209b0360576bc P 00035488ce0740898f6f92a508b01a05: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:11:25.441757 15882 raft_consensus.cc:482] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:25.441767 15879 raft_consensus.cc:482] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:25.441891 15879 raft_consensus.cc:504] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:25.441881 15882 raft_consensus.cc:504] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:25.442798 15879 leader_election.cc:288] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411), dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131:39965)
I20240501 14:11:25.442802 15882 leader_election.cc:288] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132:40939), 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411)
I20240501 14:11:25.447634 15279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d2be3d67f64fdc984578202c77ff12" candidate_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" is_pre_election: true
I20240501 14:11:25.447636 15148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d2be3d67f64fdc984578202c77ff12" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" is_pre_election: true
I20240501 14:11:25.447814 15279 raft_consensus.cc:2454] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc6733e9b3741578b9bc25604dd3bc6 in term 0.
I20240501 14:11:25.447814 15148 raft_consensus.cc:2454] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in term 0.
I20240501 14:11:25.447806 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d2be3d67f64fdc984578202c77ff12" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05" is_pre_election: true
I20240501 14:11:25.448011 14886 raft_consensus.cc:2454] T e0d2be3d67f64fdc984578202c77ff12 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in term 0.
I20240501 14:11:25.448174 15214 leader_election.cc:302] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f24c1c43aa2456eb67b2cbb9ed1a948, dcc6733e9b3741578b9bc25604dd3bc6; no voters:
I20240501 14:11:25.448171 15084 leader_election.cc:302] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f24c1c43aa2456eb67b2cbb9ed1a948, dcc6733e9b3741578b9bc25604dd3bc6; no voters:
I20240501 14:11:25.448402 15879 raft_consensus.cc:2781] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:11:25.448468 15879 raft_consensus.cc:482] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:11:25.448518 15879 raft_consensus.cc:3037] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:25.448634 15882 raft_consensus.cc:2781] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:11:25.448722 15882 raft_consensus.cc:482] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:11:25.448783 15882 raft_consensus.cc:3037] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:25.449023 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d2be3d67f64fdc984578202c77ff12" candidate_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05" is_pre_election: true
I20240501 14:11:25.449131 14886 raft_consensus.cc:2454] T e0d2be3d67f64fdc984578202c77ff12 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc6733e9b3741578b9bc25604dd3bc6 in term 0.
I20240501 14:11:25.449527 15879 raft_consensus.cc:504] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:25.449776 15882 raft_consensus.cc:504] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:25.449806 15879 leader_election.cc:288] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 election: Requested vote from peers 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411), dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131:39965)
I20240501 14:11:25.450106 15882 leader_election.cc:288] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6 [CANDIDATE]: Term 1 election: Requested vote from peers 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132:40939), 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411)
I20240501 14:11:25.450142 15148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d2be3d67f64fdc984578202c77ff12" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc6733e9b3741578b9bc25604dd3bc6"
I20240501 14:11:25.450146 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d2be3d67f64fdc984578202c77ff12" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05"
I20240501 14:11:25.450240 14886 raft_consensus.cc:3037] T e0d2be3d67f64fdc984578202c77ff12 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:25.450282 15148 raft_consensus.cc:2379] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in current term 1: Already voted for candidate dcc6733e9b3741578b9bc25604dd3bc6 in this term.
I20240501 14:11:25.450353 15279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d2be3d67f64fdc984578202c77ff12" candidate_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948"
I20240501 14:11:25.450498 15279 raft_consensus.cc:2379] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate dcc6733e9b3741578b9bc25604dd3bc6 in current term 1: Already voted for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in this term.
I20240501 14:11:25.450610 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d2be3d67f64fdc984578202c77ff12" candidate_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05"
I20240501 14:11:25.451030 15083 leader_election.cc:302] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dcc6733e9b3741578b9bc25604dd3bc6; no voters: 00035488ce0740898f6f92a508b01a05, 5f24c1c43aa2456eb67b2cbb9ed1a948
I20240501 14:11:25.451203 15882 raft_consensus.cc:2726] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 14:11:25.451238 14886 raft_consensus.cc:2454] T e0d2be3d67f64fdc984578202c77ff12 P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in term 1.
I20240501 14:11:25.451444 15214 leader_election.cc:302] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00035488ce0740898f6f92a508b01a05, 5f24c1c43aa2456eb67b2cbb9ed1a948; no voters: dcc6733e9b3741578b9bc25604dd3bc6
I20240501 14:11:25.451601 15879 raft_consensus.cc:2781] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:11:25.451786 15879 raft_consensus.cc:686] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 LEADER]: Becoming Leader. State: Replica: 5f24c1c43aa2456eb67b2cbb9ed1a948, State: Running, Role: LEADER
I20240501 14:11:25.452092 15879 consensus_queue.cc:237] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:25.454658 14743 catalog_manager.cc:5484] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } }
I20240501 14:11:25.489454 15882 raft_consensus.cc:482] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:25.489558 15882 raft_consensus.cc:504] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.489964 15882 leader_election.cc:288] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411), 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130:40469)
I20240501 14:11:25.490298 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c78677faa8a4ad28d7209b0360576bc" candidate_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05" is_pre_election: true
I20240501 14:11:25.490443 14886 raft_consensus.cc:2454] T 3c78677faa8a4ad28d7209b0360576bc P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc6733e9b3741578b9bc25604dd3bc6 in term 0.
I20240501 14:11:25.490738 15083 leader_election.cc:302] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00035488ce0740898f6f92a508b01a05, dcc6733e9b3741578b9bc25604dd3bc6; no voters:
I20240501 14:11:25.490931 15882 raft_consensus.cc:2781] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:11:25.490978 15882 raft_consensus.cc:482] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:11:25.491019 15882 raft_consensus.cc:3037] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:25.492022 15882 raft_consensus.cc:504] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.492365 15882 leader_election.cc:288] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6 [CANDIDATE]: Term 1 election: Requested vote from peers 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411), 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130:40469)
I20240501 14:11:25.492625 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c78677faa8a4ad28d7209b0360576bc" candidate_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05"
I20240501 14:11:25.492727 14886 raft_consensus.cc:3037] T 3c78677faa8a4ad28d7209b0360576bc P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:25.493460 14886 raft_consensus.cc:2454] T 3c78677faa8a4ad28d7209b0360576bc P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc6733e9b3741578b9bc25604dd3bc6 in term 1.
I20240501 14:11:25.493788 15083 leader_election.cc:302] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00035488ce0740898f6f92a508b01a05, dcc6733e9b3741578b9bc25604dd3bc6; no voters:
I20240501 14:11:25.493841 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c78677faa8a4ad28d7209b0360576bc" candidate_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f49c2edf1b54d83811340aaea48e682"
I20240501 14:11:25.493939 15882 raft_consensus.cc:2781] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:11:25.493841 15017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c78677faa8a4ad28d7209b0360576bc" candidate_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f49c2edf1b54d83811340aaea48e682" is_pre_election: true
I20240501 14:11:25.494094 15016 raft_consensus.cc:3037] T 3c78677faa8a4ad28d7209b0360576bc P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:25.494134 15882 raft_consensus.cc:686] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 LEADER]: Becoming Leader. State: Replica: dcc6733e9b3741578b9bc25604dd3bc6, State: Running, Role: LEADER
I20240501 14:11:25.494654 15882 consensus_queue.cc:237] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.495100 15016 raft_consensus.cc:2454] T 3c78677faa8a4ad28d7209b0360576bc P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc6733e9b3741578b9bc25604dd3bc6 in term 1.
I20240501 14:11:25.497417 14743 catalog_manager.cc:5484] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131). New cstate: current_term: 1 leader_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } }
W20240501 14:11:25.511987 15195 tablet.cc:2369] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:11:25.571167 15064 tablet.cc:2369] T 3c78677faa8a4ad28d7209b0360576bc P 5f49c2edf1b54d83811340aaea48e682: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:11:25.650334 15879 raft_consensus.cc:482] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:25.650475 15879 raft_consensus.cc:504] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.651008 15879 leader_election.cc:288] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411), 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130:40469)
W20240501 14:11:25.650991 14933 tablet.cc:2369] T 3c78677faa8a4ad28d7209b0360576bc P 00035488ce0740898f6f92a508b01a05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:11:25.651414 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be9fc0ba767947779aab78c6e279d25a" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05" is_pre_election: true
I20240501 14:11:25.651556 14886 raft_consensus.cc:2454] T be9fc0ba767947779aab78c6e279d25a P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in term 0.
I20240501 14:11:25.651878 15214 leader_election.cc:302] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00035488ce0740898f6f92a508b01a05, 5f24c1c43aa2456eb67b2cbb9ed1a948; no voters:
I20240501 14:11:25.652072 15879 raft_consensus.cc:2781] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:11:25.652125 15879 raft_consensus.cc:482] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:11:25.652153 15879 raft_consensus.cc:3037] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:25.652990 15879 raft_consensus.cc:504] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.653342 15879 leader_election.cc:288] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 election: Requested vote from peers 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411), 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130:40469)
I20240501 14:11:25.653692 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be9fc0ba767947779aab78c6e279d25a" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05"
I20240501 14:11:25.653795 14886 raft_consensus.cc:3037] T be9fc0ba767947779aab78c6e279d25a P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:25.654629 14886 raft_consensus.cc:2454] T be9fc0ba767947779aab78c6e279d25a P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in term 1.
I20240501 14:11:25.654628 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be9fc0ba767947779aab78c6e279d25a" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f49c2edf1b54d83811340aaea48e682" is_pre_election: true
I20240501 14:11:25.654696 15017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be9fc0ba767947779aab78c6e279d25a" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f49c2edf1b54d83811340aaea48e682"
I20240501 14:11:25.654759 15016 raft_consensus.cc:2454] T be9fc0ba767947779aab78c6e279d25a P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in term 0.
I20240501 14:11:25.654960 15214 leader_election.cc:302] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00035488ce0740898f6f92a508b01a05, 5f24c1c43aa2456eb67b2cbb9ed1a948; no voters:
I20240501 14:11:25.655105 15879 raft_consensus.cc:2781] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:11:25.655203 15879 raft_consensus.cc:686] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 LEADER]: Becoming Leader. State: Replica: 5f24c1c43aa2456eb67b2cbb9ed1a948, State: Running, Role: LEADER
I20240501 14:11:25.655341 15879 consensus_queue.cc:237] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.656877 14743 catalog_manager.cc:5484] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } }
W20240501 14:11:25.670990 15326 tablet.cc:2369] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:11:25.780325 15896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:11:25.780529 15896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:11:25.783406 15896 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 14:11:25.863211 15148 raft_consensus.cc:1261] T e0d2be3d67f64fdc984578202c77ff12 P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Refusing update from remote peer 5f24c1c43aa2456eb67b2cbb9ed1a948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:25.863919 14886 raft_consensus.cc:1261] T e0d2be3d67f64fdc984578202c77ff12 P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Refusing update from remote peer 5f24c1c43aa2456eb67b2cbb9ed1a948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:25.864521 15879 consensus_queue.cc:1035] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:25.864600 14886 raft_consensus.cc:1261] T 3c78677faa8a4ad28d7209b0360576bc P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Refusing update from remote peer dcc6733e9b3741578b9bc25604dd3bc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:25.864995 15882 consensus_queue.cc:1035] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:25.865192 15016 raft_consensus.cc:1261] T 3c78677faa8a4ad28d7209b0360576bc P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Refusing update from remote peer dcc6733e9b3741578b9bc25604dd3bc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:25.865610 15891 consensus_queue.cc:1035] T e0d2be3d67f64fdc984578202c77ff12 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:25.866518 15882 consensus_queue.cc:1035] T 3c78677faa8a4ad28d7209b0360576bc P dcc6733e9b3741578b9bc25604dd3bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:25.869280 15017 raft_consensus.cc:3037] T be9fc0ba767947779aab78c6e279d25a P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:25.870407 15017 raft_consensus.cc:1261] T be9fc0ba767947779aab78c6e279d25a P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Refusing update from remote peer 5f24c1c43aa2456eb67b2cbb9ed1a948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:25.870820 15891 consensus_queue.cc:1035] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:25.871115 14884 raft_consensus.cc:1261] T be9fc0ba767947779aab78c6e279d25a P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Refusing update from remote peer 5f24c1c43aa2456eb67b2cbb9ed1a948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:25.871470 15891 consensus_queue.cc:1035] T be9fc0ba767947779aab78c6e279d25a P 5f24c1c43aa2456eb67b2cbb9ed1a948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:25.879320 15921 mvcc.cc:205] Tried to move back new op lower bound from 7022889721309241344 to 7022889719614300160. Current Snapshot: MvccSnapshot[applied={T|T < 7022889721285926912}]
I20240501 14:11:25.880700 15935 mvcc.cc:205] Tried to move back new op lower bound from 7022889721285926912 to 7022889719614300160. Current Snapshot: MvccSnapshot[applied={T|T < 7022889721285926912}]
I20240501 14:11:25.884264 15924 mvcc.cc:205] Tried to move back new op lower bound from 7022889721295806464 to 7022889719788625920. Current Snapshot: MvccSnapshot[applied={T|T < 7022889721295806464}]
I20240501 14:11:25.899510 15933 mvcc.cc:205] Tried to move back new op lower bound from 7022889721295806464 to 7022889719788625920. Current Snapshot: MvccSnapshot[applied={T|T < 7022889721295806464}]
INSERT report
rows total: 2000
time total: 101.976 ms
time per row: 0.050988 ms
I20240501 14:11:25.933404 14736 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41026:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:11:25.938555 15128 tablet_service.cc:1467] Processing CreateTablet for tablet 31017eabdec7430e82b3ca0df3100404 (DEFAULT_TABLE table=rebalance_test_table_1 [id=8c7864bd8ca74b6ca339e1f8e3ac5cf7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:25.938843 15127 tablet_service.cc:1467] Processing CreateTablet for tablet a32067223cc64114b061db49e6696f38 (DEFAULT_TABLE table=rebalance_test_table_1 [id=8c7864bd8ca74b6ca339e1f8e3ac5cf7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:25.939126 14996 tablet_service.cc:1467] Processing CreateTablet for tablet 31017eabdec7430e82b3ca0df3100404 (DEFAULT_TABLE table=rebalance_test_table_1 [id=8c7864bd8ca74b6ca339e1f8e3ac5cf7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:25.939285 15126 tablet_service.cc:1467] Processing CreateTablet for tablet 23298c09ab7a4dce871f99929558da13 (DEFAULT_TABLE table=rebalance_test_table_1 [id=8c7864bd8ca74b6ca339e1f8e3ac5cf7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:25.939311 15259 tablet_service.cc:1467] Processing CreateTablet for tablet 31017eabdec7430e82b3ca0df3100404 (DEFAULT_TABLE table=rebalance_test_table_1 [id=8c7864bd8ca74b6ca339e1f8e3ac5cf7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:25.939371 14996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31017eabdec7430e82b3ca0df3100404. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:25.939455 14865 tablet_service.cc:1467] Processing CreateTablet for tablet 23298c09ab7a4dce871f99929558da13 (DEFAULT_TABLE table=rebalance_test_table_1 [id=8c7864bd8ca74b6ca339e1f8e3ac5cf7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:25.939545 15259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31017eabdec7430e82b3ca0df3100404. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:25.939713 14865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23298c09ab7a4dce871f99929558da13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:25.940873 14997 tablet_service.cc:1467] Processing CreateTablet for tablet a32067223cc64114b061db49e6696f38 (DEFAULT_TABLE table=rebalance_test_table_1 [id=8c7864bd8ca74b6ca339e1f8e3ac5cf7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:25.941078 14997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a32067223cc64114b061db49e6696f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:25.941759 14995 tablet_service.cc:1467] Processing CreateTablet for tablet 23298c09ab7a4dce871f99929558da13 (DEFAULT_TABLE table=rebalance_test_table_1 [id=8c7864bd8ca74b6ca339e1f8e3ac5cf7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:25.941835 15951 tablet_bootstrap.cc:492] T 31017eabdec7430e82b3ca0df3100404 P 5f49c2edf1b54d83811340aaea48e682: Bootstrap starting.
I20240501 14:11:25.938861 15128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31017eabdec7430e82b3ca0df3100404. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:25.942153 15126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23298c09ab7a4dce871f99929558da13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:25.942288 14995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23298c09ab7a4dce871f99929558da13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:25.942997 15951 tablet_bootstrap.cc:654] T 31017eabdec7430e82b3ca0df3100404 P 5f49c2edf1b54d83811340aaea48e682: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:25.944320 15954 tablet_bootstrap.cc:492] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6: Bootstrap starting.
I20240501 14:11:25.944340 15127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a32067223cc64114b061db49e6696f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:25.945279 15954 tablet_bootstrap.cc:654] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:25.945566 15952 tablet_bootstrap.cc:492] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Bootstrap starting.
I20240501 14:11:25.946300 15951 tablet_bootstrap.cc:492] T 31017eabdec7430e82b3ca0df3100404 P 5f49c2edf1b54d83811340aaea48e682: No bootstrap required, opened a new log
I20240501 14:11:25.946337 15952 tablet_bootstrap.cc:654] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:25.946377 15951 ts_tablet_manager.cc:1367] T 31017eabdec7430e82b3ca0df3100404 P 5f49c2edf1b54d83811340aaea48e682: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:11:25.946769 15955 tablet_bootstrap.cc:492] T 23298c09ab7a4dce871f99929558da13 P 00035488ce0740898f6f92a508b01a05: Bootstrap starting.
I20240501 14:11:25.946828 15951 raft_consensus.cc:348] T 31017eabdec7430e82b3ca0df3100404 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:25.946930 15951 raft_consensus.cc:374] T 31017eabdec7430e82b3ca0df3100404 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:25.946964 15951 raft_consensus.cc:729] T 31017eabdec7430e82b3ca0df3100404 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f49c2edf1b54d83811340aaea48e682, State: Initialized, Role: FOLLOWER
I20240501 14:11:25.947082 15951 consensus_queue.cc:260] T 31017eabdec7430e82b3ca0df3100404 P 5f49c2edf1b54d83811340aaea48e682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:25.947470 15951 ts_tablet_manager.cc:1398] T 31017eabdec7430e82b3ca0df3100404 P 5f49c2edf1b54d83811340aaea48e682: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:25.947563 15955 tablet_bootstrap.cc:654] T 23298c09ab7a4dce871f99929558da13 P 00035488ce0740898f6f92a508b01a05: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:25.947625 15951 tablet_bootstrap.cc:492] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682: Bootstrap starting.
I20240501 14:11:25.947937 15259 tablet_service.cc:1467] Processing CreateTablet for tablet a32067223cc64114b061db49e6696f38 (DEFAULT_TABLE table=rebalance_test_table_1 [id=8c7864bd8ca74b6ca339e1f8e3ac5cf7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:25.948174 15259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a32067223cc64114b061db49e6696f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:25.948506 15951 tablet_bootstrap.cc:654] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:25.948518 15952 tablet_bootstrap.cc:492] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948: No bootstrap required, opened a new log
I20240501 14:11:25.948590 15952 ts_tablet_manager.cc:1367] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:11:25.948999 15952 raft_consensus.cc:348] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:25.949110 15952 raft_consensus.cc:374] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:25.949157 15952 raft_consensus.cc:729] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f24c1c43aa2456eb67b2cbb9ed1a948, State: Initialized, Role: FOLLOWER
I20240501 14:11:25.949285 15952 consensus_queue.cc:260] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:25.949694 15952 ts_tablet_manager.cc:1398] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:25.950359 15955 tablet_bootstrap.cc:492] T 23298c09ab7a4dce871f99929558da13 P 00035488ce0740898f6f92a508b01a05: No bootstrap required, opened a new log
I20240501 14:11:25.950428 15955 ts_tablet_manager.cc:1367] T 23298c09ab7a4dce871f99929558da13 P 00035488ce0740898f6f92a508b01a05: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:11:25.950740 15955 raft_consensus.cc:348] T 23298c09ab7a4dce871f99929558da13 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.950815 15955 raft_consensus.cc:374] T 23298c09ab7a4dce871f99929558da13 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:25.950834 15955 raft_consensus.cc:729] T 23298c09ab7a4dce871f99929558da13 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00035488ce0740898f6f92a508b01a05, State: Initialized, Role: FOLLOWER
I20240501 14:11:25.950934 15955 consensus_queue.cc:260] T 23298c09ab7a4dce871f99929558da13 P 00035488ce0740898f6f92a508b01a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.951179 15955 ts_tablet_manager.cc:1398] T 23298c09ab7a4dce871f99929558da13 P 00035488ce0740898f6f92a508b01a05: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:11:25.951529 15951 tablet_bootstrap.cc:492] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682: No bootstrap required, opened a new log
I20240501 14:11:25.951610 15951 ts_tablet_manager.cc:1367] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:11:25.951984 15951 raft_consensus.cc:348] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:25.952083 15951 raft_consensus.cc:374] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:25.952116 15951 raft_consensus.cc:729] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f49c2edf1b54d83811340aaea48e682, State: Initialized, Role: FOLLOWER
I20240501 14:11:25.952265 15951 consensus_queue.cc:260] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:25.952612 15951 ts_tablet_manager.cc:1398] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:25.952816 15951 tablet_bootstrap.cc:492] T 23298c09ab7a4dce871f99929558da13 P 5f49c2edf1b54d83811340aaea48e682: Bootstrap starting.
I20240501 14:11:25.953627 15951 tablet_bootstrap.cc:654] T 23298c09ab7a4dce871f99929558da13 P 5f49c2edf1b54d83811340aaea48e682: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:25.954145 15952 tablet_bootstrap.cc:492] T a32067223cc64114b061db49e6696f38 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Bootstrap starting.
I20240501 14:11:25.954715 15954 tablet_bootstrap.cc:492] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6: No bootstrap required, opened a new log
I20240501 14:11:25.954807 15954 ts_tablet_manager.cc:1367] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240501 14:11:25.954978 15952 tablet_bootstrap.cc:654] T a32067223cc64114b061db49e6696f38 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:25.954990 15951 tablet_bootstrap.cc:492] T 23298c09ab7a4dce871f99929558da13 P 5f49c2edf1b54d83811340aaea48e682: No bootstrap required, opened a new log
I20240501 14:11:25.955045 15951 ts_tablet_manager.cc:1367] T 23298c09ab7a4dce871f99929558da13 P 5f49c2edf1b54d83811340aaea48e682: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:11:25.955269 15954 raft_consensus.cc:348] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.955364 15954 raft_consensus.cc:374] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:25.955400 15954 raft_consensus.cc:729] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc6733e9b3741578b9bc25604dd3bc6, State: Initialized, Role: FOLLOWER
I20240501 14:11:25.955389 15951 raft_consensus.cc:348] T 23298c09ab7a4dce871f99929558da13 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.955484 15951 raft_consensus.cc:374] T 23298c09ab7a4dce871f99929558da13 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:25.955521 15951 raft_consensus.cc:729] T 23298c09ab7a4dce871f99929558da13 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f49c2edf1b54d83811340aaea48e682, State: Initialized, Role: FOLLOWER
I20240501 14:11:25.955552 15954 consensus_queue.cc:260] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.955638 15951 consensus_queue.cc:260] T 23298c09ab7a4dce871f99929558da13 P 5f49c2edf1b54d83811340aaea48e682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.955971 15951 ts_tablet_manager.cc:1398] T 23298c09ab7a4dce871f99929558da13 P 5f49c2edf1b54d83811340aaea48e682: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:25.956394 15954 ts_tablet_manager.cc:1398] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:11:25.956559 15954 tablet_bootstrap.cc:492] T 31017eabdec7430e82b3ca0df3100404 P dcc6733e9b3741578b9bc25604dd3bc6: Bootstrap starting.
I20240501 14:11:25.956665 15952 tablet_bootstrap.cc:492] T a32067223cc64114b061db49e6696f38 P 5f24c1c43aa2456eb67b2cbb9ed1a948: No bootstrap required, opened a new log
I20240501 14:11:25.956745 15952 ts_tablet_manager.cc:1367] T a32067223cc64114b061db49e6696f38 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:11:25.957197 15952 raft_consensus.cc:348] T a32067223cc64114b061db49e6696f38 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:25.957306 15952 raft_consensus.cc:374] T a32067223cc64114b061db49e6696f38 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:25.957335 15952 raft_consensus.cc:729] T a32067223cc64114b061db49e6696f38 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f24c1c43aa2456eb67b2cbb9ed1a948, State: Initialized, Role: FOLLOWER
I20240501 14:11:25.957502 15952 consensus_queue.cc:260] T a32067223cc64114b061db49e6696f38 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:25.957749 15954 tablet_bootstrap.cc:654] T 31017eabdec7430e82b3ca0df3100404 P dcc6733e9b3741578b9bc25604dd3bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:25.957782 15952 ts_tablet_manager.cc:1398] T a32067223cc64114b061db49e6696f38 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:25.959018 15954 tablet_bootstrap.cc:492] T 31017eabdec7430e82b3ca0df3100404 P dcc6733e9b3741578b9bc25604dd3bc6: No bootstrap required, opened a new log
I20240501 14:11:25.959079 15954 ts_tablet_manager.cc:1367] T 31017eabdec7430e82b3ca0df3100404 P dcc6733e9b3741578b9bc25604dd3bc6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:11:25.959498 15954 raft_consensus.cc:348] T 31017eabdec7430e82b3ca0df3100404 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:25.959600 15954 raft_consensus.cc:374] T 31017eabdec7430e82b3ca0df3100404 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:25.959632 15954 raft_consensus.cc:729] T 31017eabdec7430e82b3ca0df3100404 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc6733e9b3741578b9bc25604dd3bc6, State: Initialized, Role: FOLLOWER
I20240501 14:11:25.959739 15954 consensus_queue.cc:260] T 31017eabdec7430e82b3ca0df3100404 P dcc6733e9b3741578b9bc25604dd3bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:25.959961 15954 ts_tablet_manager.cc:1398] T 31017eabdec7430e82b3ca0df3100404 P dcc6733e9b3741578b9bc25604dd3bc6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:11:25.960109 15954 tablet_bootstrap.cc:492] T a32067223cc64114b061db49e6696f38 P dcc6733e9b3741578b9bc25604dd3bc6: Bootstrap starting.
I20240501 14:11:25.960744 15954 tablet_bootstrap.cc:654] T a32067223cc64114b061db49e6696f38 P dcc6733e9b3741578b9bc25604dd3bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:25.961755 15954 tablet_bootstrap.cc:492] T a32067223cc64114b061db49e6696f38 P dcc6733e9b3741578b9bc25604dd3bc6: No bootstrap required, opened a new log
I20240501 14:11:25.961845 15954 ts_tablet_manager.cc:1367] T a32067223cc64114b061db49e6696f38 P dcc6733e9b3741578b9bc25604dd3bc6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:11:25.962287 15954 raft_consensus.cc:348] T a32067223cc64114b061db49e6696f38 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:25.962394 15954 raft_consensus.cc:374] T a32067223cc64114b061db49e6696f38 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:25.962440 15954 raft_consensus.cc:729] T a32067223cc64114b061db49e6696f38 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc6733e9b3741578b9bc25604dd3bc6, State: Initialized, Role: FOLLOWER
I20240501 14:11:25.962546 15954 consensus_queue.cc:260] T a32067223cc64114b061db49e6696f38 P dcc6733e9b3741578b9bc25604dd3bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:25.962853 15954 ts_tablet_manager.cc:1398] T a32067223cc64114b061db49e6696f38 P dcc6733e9b3741578b9bc25604dd3bc6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:11:25.967757 15893 raft_consensus.cc:482] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:25.967834 15893 raft_consensus.cc:504] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.968168 15893 leader_election.cc:288] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411), 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130:40469)
I20240501 14:11:25.968482 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23298c09ab7a4dce871f99929558da13" candidate_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05" is_pre_election: true
I20240501 14:11:25.968523 15017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23298c09ab7a4dce871f99929558da13" candidate_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f49c2edf1b54d83811340aaea48e682" is_pre_election: true
I20240501 14:11:25.968602 14884 raft_consensus.cc:2454] T 23298c09ab7a4dce871f99929558da13 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc6733e9b3741578b9bc25604dd3bc6 in term 0.
I20240501 14:11:25.968618 15017 raft_consensus.cc:2454] T 23298c09ab7a4dce871f99929558da13 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc6733e9b3741578b9bc25604dd3bc6 in term 0.
I20240501 14:11:25.968859 15083 leader_election.cc:302] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00035488ce0740898f6f92a508b01a05, dcc6733e9b3741578b9bc25604dd3bc6; no voters:
I20240501 14:11:25.968997 15893 raft_consensus.cc:2781] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:11:25.969051 15893 raft_consensus.cc:482] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:11:25.969077 15893 raft_consensus.cc:3037] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:25.969949 15893 raft_consensus.cc:504] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.970227 15893 leader_election.cc:288] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6 [CANDIDATE]: Term 1 election: Requested vote from peers 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411), 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130:40469)
I20240501 14:11:25.970501 15017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23298c09ab7a4dce871f99929558da13" candidate_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f49c2edf1b54d83811340aaea48e682"
I20240501 14:11:25.970489 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23298c09ab7a4dce871f99929558da13" candidate_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05"
I20240501 14:11:25.970599 14884 raft_consensus.cc:3037] T 23298c09ab7a4dce871f99929558da13 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:25.970616 15017 raft_consensus.cc:3037] T 23298c09ab7a4dce871f99929558da13 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:25.971467 14884 raft_consensus.cc:2454] T 23298c09ab7a4dce871f99929558da13 P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc6733e9b3741578b9bc25604dd3bc6 in term 1.
I20240501 14:11:25.971467 15017 raft_consensus.cc:2454] T 23298c09ab7a4dce871f99929558da13 P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc6733e9b3741578b9bc25604dd3bc6 in term 1.
I20240501 14:11:25.971750 15081 leader_election.cc:302] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f49c2edf1b54d83811340aaea48e682, dcc6733e9b3741578b9bc25604dd3bc6; no voters:
I20240501 14:11:25.971907 15893 raft_consensus.cc:2781] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:11:25.971982 15893 raft_consensus.cc:686] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 LEADER]: Becoming Leader. State: Replica: dcc6733e9b3741578b9bc25604dd3bc6, State: Running, Role: LEADER
I20240501 14:11:25.972101 15893 consensus_queue.cc:237] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:25.973320 14738 catalog_manager.cc:5484] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131). New cstate: current_term: 1 leader_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } }
I20240501 14:11:26.080300 15876 raft_consensus.cc:482] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:26.080430 15876 raft_consensus.cc:504] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:26.081431 15876 leader_election.cc:288] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131:39965), 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132:40939)
I20240501 14:11:26.084957 15148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32067223cc64114b061db49e6696f38" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" is_pre_election: true
I20240501 14:11:26.085004 15279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32067223cc64114b061db49e6696f38" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" is_pre_election: true
I20240501 14:11:26.085088 15148 raft_consensus.cc:2454] T a32067223cc64114b061db49e6696f38 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 0.
I20240501 14:11:26.085132 15279 raft_consensus.cc:2454] T a32067223cc64114b061db49e6696f38 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 0.
I20240501 14:11:26.085402 14952 leader_election.cc:302] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f49c2edf1b54d83811340aaea48e682, dcc6733e9b3741578b9bc25604dd3bc6; no voters:
I20240501 14:11:26.085645 15876 raft_consensus.cc:2781] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:11:26.085701 15876 raft_consensus.cc:482] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:11:26.085753 15876 raft_consensus.cc:3037] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:26.086544 15876 raft_consensus.cc:504] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:26.086863 15876 leader_election.cc:288] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 election: Requested vote from peers dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131:39965), 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132:40939)
I20240501 14:11:26.087157 15279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32067223cc64114b061db49e6696f38" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948"
I20240501 14:11:26.087253 15279 raft_consensus.cc:3037] T a32067223cc64114b061db49e6696f38 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:26.087234 15148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32067223cc64114b061db49e6696f38" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc6733e9b3741578b9bc25604dd3bc6"
I20240501 14:11:26.087337 15148 raft_consensus.cc:3037] T a32067223cc64114b061db49e6696f38 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:26.087986 15279 raft_consensus.cc:2454] T a32067223cc64114b061db49e6696f38 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 1.
I20240501 14:11:26.088107 15148 raft_consensus.cc:2454] T a32067223cc64114b061db49e6696f38 P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 1.
I20240501 14:11:26.088299 14953 leader_election.cc:302] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f24c1c43aa2456eb67b2cbb9ed1a948, 5f49c2edf1b54d83811340aaea48e682; no voters:
I20240501 14:11:26.088454 15876 raft_consensus.cc:2781] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:11:26.088656 15876 raft_consensus.cc:686] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682 [term 1 LEADER]: Becoming Leader. State: Replica: 5f49c2edf1b54d83811340aaea48e682, State: Running, Role: LEADER
I20240501 14:11:26.088923 15876 consensus_queue.cc:237] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:26.090610 14738 catalog_manager.cc:5484] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "5f49c2edf1b54d83811340aaea48e682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } }
I20240501 14:11:26.099689 15891 raft_consensus.cc:482] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:26.099781 15891 raft_consensus.cc:504] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:26.100116 15891 leader_election.cc:288] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130:40469), dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131:39965)
I20240501 14:11:26.100445 15017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31017eabdec7430e82b3ca0df3100404" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f49c2edf1b54d83811340aaea48e682" is_pre_election: true
I20240501 14:11:26.100476 15148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31017eabdec7430e82b3ca0df3100404" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" is_pre_election: true
I20240501 14:11:26.100571 15148 raft_consensus.cc:2454] T 31017eabdec7430e82b3ca0df3100404 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in term 0.
I20240501 14:11:26.100574 15017 raft_consensus.cc:2454] T 31017eabdec7430e82b3ca0df3100404 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in term 0.
I20240501 14:11:26.100803 15214 leader_election.cc:302] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f24c1c43aa2456eb67b2cbb9ed1a948, dcc6733e9b3741578b9bc25604dd3bc6; no voters:
I20240501 14:11:26.100952 15891 raft_consensus.cc:2781] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:11:26.101009 15891 raft_consensus.cc:482] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:11:26.101037 15891 raft_consensus.cc:3037] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:26.101759 15891 raft_consensus.cc:504] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:26.102028 15891 leader_election.cc:288] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 election: Requested vote from peers 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130:40469), dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131:39965)
I20240501 14:11:26.102315 15148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31017eabdec7430e82b3ca0df3100404" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc6733e9b3741578b9bc25604dd3bc6"
I20240501 14:11:26.102337 15017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31017eabdec7430e82b3ca0df3100404" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f49c2edf1b54d83811340aaea48e682"
I20240501 14:11:26.102413 15017 raft_consensus.cc:3037] T 31017eabdec7430e82b3ca0df3100404 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:26.102404 15148 raft_consensus.cc:3037] T 31017eabdec7430e82b3ca0df3100404 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:26.103049 15017 raft_consensus.cc:2454] T 31017eabdec7430e82b3ca0df3100404 P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in term 1.
I20240501 14:11:26.103094 15148 raft_consensus.cc:2454] T 31017eabdec7430e82b3ca0df3100404 P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in term 1.
I20240501 14:11:26.103339 15214 leader_election.cc:302] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f24c1c43aa2456eb67b2cbb9ed1a948, dcc6733e9b3741578b9bc25604dd3bc6; no voters:
I20240501 14:11:26.103492 15891 raft_consensus.cc:2781] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:11:26.103576 15891 raft_consensus.cc:686] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 LEADER]: Becoming Leader. State: Replica: 5f24c1c43aa2456eb67b2cbb9ed1a948, State: Running, Role: LEADER
I20240501 14:11:26.103734 15891 consensus_queue.cc:237] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:26.105033 14738 catalog_manager.cc:5484] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } }
W20240501 14:11:26.232738 15970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:11:26.232985 15970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:11:26.236346 15970 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 14:11:26.312935 15148 raft_consensus.cc:1261] T 31017eabdec7430e82b3ca0df3100404 P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Refusing update from remote peer 5f24c1c43aa2456eb67b2cbb9ed1a948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:26.314253 15891 consensus_queue.cc:1035] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:26.314708 15017 raft_consensus.cc:1261] T 31017eabdec7430e82b3ca0df3100404 P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Refusing update from remote peer 5f24c1c43aa2456eb67b2cbb9ed1a948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:26.315268 15891 consensus_queue.cc:1035] T 31017eabdec7430e82b3ca0df3100404 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:26.316138 15279 raft_consensus.cc:1261] T a32067223cc64114b061db49e6696f38 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Refusing update from remote peer 5f49c2edf1b54d83811340aaea48e682: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:26.317236 15967 consensus_queue.cc:1035] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:26.319481 14884 raft_consensus.cc:1261] T 23298c09ab7a4dce871f99929558da13 P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Refusing update from remote peer dcc6733e9b3741578b9bc25604dd3bc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:26.319905 15882 consensus_queue.cc:1035] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:26.328691 15016 raft_consensus.cc:1261] T 23298c09ab7a4dce871f99929558da13 P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Refusing update from remote peer dcc6733e9b3741578b9bc25604dd3bc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:26.329600 15893 consensus_queue.cc:1035] T 23298c09ab7a4dce871f99929558da13 P dcc6733e9b3741578b9bc25604dd3bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:26.346135 15148 raft_consensus.cc:1261] T a32067223cc64114b061db49e6696f38 P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Refusing update from remote peer 5f49c2edf1b54d83811340aaea48e682: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:26.346500 15876 consensus_queue.cc:1035] T a32067223cc64114b061db49e6696f38 P 5f49c2edf1b54d83811340aaea48e682 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 107.499 ms
time per row: 0.0537497 ms
I20240501 14:11:26.384428 14744 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41026:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:11:26.389623 14865 tablet_service.cc:1467] Processing CreateTablet for tablet bae4a31367f746bba17d2540cd36c4d5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=56b83aa045104131ad9d8783384eaa80]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:26.389741 15259 tablet_service.cc:1467] Processing CreateTablet for tablet bae4a31367f746bba17d2540cd36c4d5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=56b83aa045104131ad9d8783384eaa80]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:26.389956 14865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bae4a31367f746bba17d2540cd36c4d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:26.390038 15259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bae4a31367f746bba17d2540cd36c4d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:26.390619 14997 tablet_service.cc:1467] Processing CreateTablet for tablet a99203e550254896a5e269c91201fa85 (DEFAULT_TABLE table=rebalance_test_table_2 [id=56b83aa045104131ad9d8783384eaa80]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:26.390614 14995 tablet_service.cc:1467] Processing CreateTablet for tablet bae4a31367f746bba17d2540cd36c4d5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=56b83aa045104131ad9d8783384eaa80]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:26.390900 14997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a99203e550254896a5e269c91201fa85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:26.391841 15952 tablet_bootstrap.cc:492] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Bootstrap starting.
I20240501 14:11:26.392220 15258 tablet_service.cc:1467] Processing CreateTablet for tablet a99203e550254896a5e269c91201fa85 (DEFAULT_TABLE table=rebalance_test_table_2 [id=56b83aa045104131ad9d8783384eaa80]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:26.392457 15258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a99203e550254896a5e269c91201fa85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:26.392725 15952 tablet_bootstrap.cc:654] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:26.392904 15955 tablet_bootstrap.cc:492] T bae4a31367f746bba17d2540cd36c4d5 P 00035488ce0740898f6f92a508b01a05: Bootstrap starting.
I20240501 14:11:26.393677 15955 tablet_bootstrap.cc:654] T bae4a31367f746bba17d2540cd36c4d5 P 00035488ce0740898f6f92a508b01a05: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:26.393816 15952 tablet_bootstrap.cc:492] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948: No bootstrap required, opened a new log
I20240501 14:11:26.393903 15952 ts_tablet_manager.cc:1367] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:11:26.394325 15952 raft_consensus.cc:348] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:26.394438 15952 raft_consensus.cc:374] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:26.394480 15952 raft_consensus.cc:729] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f24c1c43aa2456eb67b2cbb9ed1a948, State: Initialized, Role: FOLLOWER
I20240501 14:11:26.394593 15952 consensus_queue.cc:260] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:26.394783 14865 tablet_service.cc:1467] Processing CreateTablet for tablet 9713eefbff264eb592a424e4fbc5000c (DEFAULT_TABLE table=rebalance_test_table_2 [id=56b83aa045104131ad9d8783384eaa80]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:26.394902 15952 ts_tablet_manager.cc:1398] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:26.394997 14865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9713eefbff264eb592a424e4fbc5000c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:26.394783 14866 tablet_service.cc:1467] Processing CreateTablet for tablet a99203e550254896a5e269c91201fa85 (DEFAULT_TABLE table=rebalance_test_table_2 [id=56b83aa045104131ad9d8783384eaa80]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:26.395066 15952 tablet_bootstrap.cc:492] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Bootstrap starting.
I20240501 14:11:26.395134 14866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a99203e550254896a5e269c91201fa85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:26.395267 15127 tablet_service.cc:1467] Processing CreateTablet for tablet 9713eefbff264eb592a424e4fbc5000c (DEFAULT_TABLE table=rebalance_test_table_2 [id=56b83aa045104131ad9d8783384eaa80]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:26.395535 15127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9713eefbff264eb592a424e4fbc5000c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:26.395916 15952 tablet_bootstrap.cc:654] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:26.397537 15954 tablet_bootstrap.cc:492] T 9713eefbff264eb592a424e4fbc5000c P dcc6733e9b3741578b9bc25604dd3bc6: Bootstrap starting.
I20240501 14:11:26.397619 15951 tablet_bootstrap.cc:492] T a99203e550254896a5e269c91201fa85 P 5f49c2edf1b54d83811340aaea48e682: Bootstrap starting.
I20240501 14:11:26.397771 15259 tablet_service.cc:1467] Processing CreateTablet for tablet 9713eefbff264eb592a424e4fbc5000c (DEFAULT_TABLE table=rebalance_test_table_2 [id=56b83aa045104131ad9d8783384eaa80]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:26.397991 15259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9713eefbff264eb592a424e4fbc5000c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:26.398388 15954 tablet_bootstrap.cc:654] T 9713eefbff264eb592a424e4fbc5000c P dcc6733e9b3741578b9bc25604dd3bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:26.398454 15951 tablet_bootstrap.cc:654] T a99203e550254896a5e269c91201fa85 P 5f49c2edf1b54d83811340aaea48e682: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:26.399247 14995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bae4a31367f746bba17d2540cd36c4d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:26.401021 15952 tablet_bootstrap.cc:492] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948: No bootstrap required, opened a new log
I20240501 14:11:26.401149 15954 tablet_bootstrap.cc:492] T 9713eefbff264eb592a424e4fbc5000c P dcc6733e9b3741578b9bc25604dd3bc6: No bootstrap required, opened a new log
I20240501 14:11:26.401149 15952 ts_tablet_manager.cc:1367] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:11:26.401230 15954 ts_tablet_manager.cc:1367] T 9713eefbff264eb592a424e4fbc5000c P dcc6733e9b3741578b9bc25604dd3bc6: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:11:26.401310 15955 tablet_bootstrap.cc:492] T bae4a31367f746bba17d2540cd36c4d5 P 00035488ce0740898f6f92a508b01a05: No bootstrap required, opened a new log
I20240501 14:11:26.401405 15955 ts_tablet_manager.cc:1367] T bae4a31367f746bba17d2540cd36c4d5 P 00035488ce0740898f6f92a508b01a05: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 14:11:26.401630 15954 raft_consensus.cc:348] T 9713eefbff264eb592a424e4fbc5000c P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } }
I20240501 14:11:26.401729 15954 raft_consensus.cc:374] T 9713eefbff264eb592a424e4fbc5000c P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:26.401782 15954 raft_consensus.cc:729] T 9713eefbff264eb592a424e4fbc5000c P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc6733e9b3741578b9bc25604dd3bc6, State: Initialized, Role: FOLLOWER
I20240501 14:11:26.401830 15955 raft_consensus.cc:348] T bae4a31367f746bba17d2540cd36c4d5 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:26.401943 15955 raft_consensus.cc:374] T bae4a31367f746bba17d2540cd36c4d5 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:26.401917 15954 consensus_queue.cc:260] T 9713eefbff264eb592a424e4fbc5000c P dcc6733e9b3741578b9bc25604dd3bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } }
I20240501 14:11:26.401989 15955 raft_consensus.cc:729] T bae4a31367f746bba17d2540cd36c4d5 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00035488ce0740898f6f92a508b01a05, State: Initialized, Role: FOLLOWER
I20240501 14:11:26.402092 15952 raft_consensus.cc:348] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:26.402135 15955 consensus_queue.cc:260] T bae4a31367f746bba17d2540cd36c4d5 P 00035488ce0740898f6f92a508b01a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:26.402204 15954 ts_tablet_manager.cc:1398] T 9713eefbff264eb592a424e4fbc5000c P dcc6733e9b3741578b9bc25604dd3bc6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:26.402292 15952 raft_consensus.cc:374] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:26.402369 15952 raft_consensus.cc:729] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f24c1c43aa2456eb67b2cbb9ed1a948, State: Initialized, Role: FOLLOWER
I20240501 14:11:26.402426 15955 ts_tablet_manager.cc:1398] T bae4a31367f746bba17d2540cd36c4d5 P 00035488ce0740898f6f92a508b01a05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:26.402530 15952 consensus_queue.cc:260] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:26.402822 15952 ts_tablet_manager.cc:1398] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:11:26.402974 15952 tablet_bootstrap.cc:492] T 9713eefbff264eb592a424e4fbc5000c P 5f24c1c43aa2456eb67b2cbb9ed1a948: Bootstrap starting.
I20240501 14:11:26.402594 15955 tablet_bootstrap.cc:492] T a99203e550254896a5e269c91201fa85 P 00035488ce0740898f6f92a508b01a05: Bootstrap starting.
I20240501 14:11:26.404000 15952 tablet_bootstrap.cc:654] T 9713eefbff264eb592a424e4fbc5000c P 5f24c1c43aa2456eb67b2cbb9ed1a948: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:26.405278 15955 tablet_bootstrap.cc:654] T a99203e550254896a5e269c91201fa85 P 00035488ce0740898f6f92a508b01a05: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:26.406488 15955 tablet_bootstrap.cc:492] T a99203e550254896a5e269c91201fa85 P 00035488ce0740898f6f92a508b01a05: No bootstrap required, opened a new log
I20240501 14:11:26.406566 15955 ts_tablet_manager.cc:1367] T a99203e550254896a5e269c91201fa85 P 00035488ce0740898f6f92a508b01a05: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:11:26.406908 15955 raft_consensus.cc:348] T a99203e550254896a5e269c91201fa85 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:26.407001 15955 raft_consensus.cc:374] T a99203e550254896a5e269c91201fa85 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:26.407032 15955 raft_consensus.cc:729] T a99203e550254896a5e269c91201fa85 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00035488ce0740898f6f92a508b01a05, State: Initialized, Role: FOLLOWER
I20240501 14:11:26.407161 15955 consensus_queue.cc:260] T a99203e550254896a5e269c91201fa85 P 00035488ce0740898f6f92a508b01a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:26.407802 15955 ts_tablet_manager.cc:1398] T a99203e550254896a5e269c91201fa85 P 00035488ce0740898f6f92a508b01a05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:26.407970 15955 tablet_bootstrap.cc:492] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05: Bootstrap starting.
I20240501 14:11:26.408841 15955 tablet_bootstrap.cc:654] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:26.409896 15955 tablet_bootstrap.cc:492] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05: No bootstrap required, opened a new log
I20240501 14:11:26.409976 15955 ts_tablet_manager.cc:1367] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:11:26.410358 15955 raft_consensus.cc:348] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } }
I20240501 14:11:26.410465 15955 raft_consensus.cc:374] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:26.410507 15955 raft_consensus.cc:729] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00035488ce0740898f6f92a508b01a05, State: Initialized, Role: FOLLOWER
I20240501 14:11:26.410630 15955 consensus_queue.cc:260] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } }
I20240501 14:11:26.410880 15955 ts_tablet_manager.cc:1398] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:11:26.411420 15951 tablet_bootstrap.cc:492] T a99203e550254896a5e269c91201fa85 P 5f49c2edf1b54d83811340aaea48e682: No bootstrap required, opened a new log
I20240501 14:11:26.411499 15951 ts_tablet_manager.cc:1367] T a99203e550254896a5e269c91201fa85 P 5f49c2edf1b54d83811340aaea48e682: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:11:26.411630 15952 tablet_bootstrap.cc:492] T 9713eefbff264eb592a424e4fbc5000c P 5f24c1c43aa2456eb67b2cbb9ed1a948: No bootstrap required, opened a new log
I20240501 14:11:26.411726 15952 ts_tablet_manager.cc:1367] T 9713eefbff264eb592a424e4fbc5000c P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:11:26.411978 15951 raft_consensus.cc:348] T a99203e550254896a5e269c91201fa85 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:26.412070 15951 raft_consensus.cc:374] T a99203e550254896a5e269c91201fa85 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:26.412096 15951 raft_consensus.cc:729] T a99203e550254896a5e269c91201fa85 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f49c2edf1b54d83811340aaea48e682, State: Initialized, Role: FOLLOWER
I20240501 14:11:26.412112 15952 raft_consensus.cc:348] T 9713eefbff264eb592a424e4fbc5000c P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } }
I20240501 14:11:26.412201 15952 raft_consensus.cc:374] T 9713eefbff264eb592a424e4fbc5000c P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:26.412204 15951 consensus_queue.cc:260] T a99203e550254896a5e269c91201fa85 P 5f49c2edf1b54d83811340aaea48e682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:26.412267 15952 raft_consensus.cc:729] T 9713eefbff264eb592a424e4fbc5000c P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f24c1c43aa2456eb67b2cbb9ed1a948, State: Initialized, Role: FOLLOWER
I20240501 14:11:26.412396 15952 consensus_queue.cc:260] T 9713eefbff264eb592a424e4fbc5000c P 5f24c1c43aa2456eb67b2cbb9ed1a948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } }
I20240501 14:11:26.412506 15951 ts_tablet_manager.cc:1398] T a99203e550254896a5e269c91201fa85 P 5f49c2edf1b54d83811340aaea48e682: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:26.412676 15951 tablet_bootstrap.cc:492] T bae4a31367f746bba17d2540cd36c4d5 P 5f49c2edf1b54d83811340aaea48e682: Bootstrap starting.
I20240501 14:11:26.412694 15952 ts_tablet_manager.cc:1398] T 9713eefbff264eb592a424e4fbc5000c P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:26.413558 15951 tablet_bootstrap.cc:654] T bae4a31367f746bba17d2540cd36c4d5 P 5f49c2edf1b54d83811340aaea48e682: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:26.414921 15951 tablet_bootstrap.cc:492] T bae4a31367f746bba17d2540cd36c4d5 P 5f49c2edf1b54d83811340aaea48e682: No bootstrap required, opened a new log
I20240501 14:11:26.415007 15951 ts_tablet_manager.cc:1367] T bae4a31367f746bba17d2540cd36c4d5 P 5f49c2edf1b54d83811340aaea48e682: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:11:26.415645 15951 raft_consensus.cc:348] T bae4a31367f746bba17d2540cd36c4d5 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:26.415724 15951 raft_consensus.cc:374] T bae4a31367f746bba17d2540cd36c4d5 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:26.415766 15951 raft_consensus.cc:729] T bae4a31367f746bba17d2540cd36c4d5 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f49c2edf1b54d83811340aaea48e682, State: Initialized, Role: FOLLOWER
I20240501 14:11:26.415886 15951 consensus_queue.cc:260] T bae4a31367f746bba17d2540cd36c4d5 P 5f49c2edf1b54d83811340aaea48e682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:26.416153 15951 ts_tablet_manager.cc:1398] T bae4a31367f746bba17d2540cd36c4d5 P 5f49c2edf1b54d83811340aaea48e682: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:26.434556 15999 raft_consensus.cc:482] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:26.434656 15999 raft_consensus.cc:504] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:26.435073 15999 leader_election.cc:288] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411), 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130:40469)
I20240501 14:11:26.435403 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae4a31367f746bba17d2540cd36c4d5" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05" is_pre_election: true
I20240501 14:11:26.435428 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae4a31367f746bba17d2540cd36c4d5" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f49c2edf1b54d83811340aaea48e682" is_pre_election: true
I20240501 14:11:26.435539 14884 raft_consensus.cc:2454] T bae4a31367f746bba17d2540cd36c4d5 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in term 0.
I20240501 14:11:26.435549 15016 raft_consensus.cc:2454] T bae4a31367f746bba17d2540cd36c4d5 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in term 0.
I20240501 14:11:26.435810 15212 leader_election.cc:302] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f24c1c43aa2456eb67b2cbb9ed1a948, 5f49c2edf1b54d83811340aaea48e682; no voters:
I20240501 14:11:26.435984 15999 raft_consensus.cc:2781] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:11:26.436038 15999 raft_consensus.cc:482] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:11:26.436064 15999 raft_consensus.cc:3037] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:26.436832 15999 raft_consensus.cc:504] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:26.437144 15999 leader_election.cc:288] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 election: Requested vote from peers 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411), 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130:40469)
I20240501 14:11:26.437424 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae4a31367f746bba17d2540cd36c4d5" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05"
I20240501 14:11:26.437530 14884 raft_consensus.cc:3037] T bae4a31367f746bba17d2540cd36c4d5 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:26.437494 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae4a31367f746bba17d2540cd36c4d5" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f49c2edf1b54d83811340aaea48e682"
I20240501 14:11:26.437614 15016 raft_consensus.cc:3037] T bae4a31367f746bba17d2540cd36c4d5 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:26.438494 14884 raft_consensus.cc:2454] T bae4a31367f746bba17d2540cd36c4d5 P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in term 1.
I20240501 14:11:26.438516 15016 raft_consensus.cc:2454] T bae4a31367f746bba17d2540cd36c4d5 P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in term 1.
I20240501 14:11:26.438799 15212 leader_election.cc:302] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f24c1c43aa2456eb67b2cbb9ed1a948, 5f49c2edf1b54d83811340aaea48e682; no voters:
I20240501 14:11:26.438988 15999 raft_consensus.cc:2781] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:11:26.439098 15999 raft_consensus.cc:686] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 LEADER]: Becoming Leader. State: Replica: 5f24c1c43aa2456eb67b2cbb9ed1a948, State: Running, Role: LEADER
I20240501 14:11:26.439235 15999 consensus_queue.cc:237] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:26.440580 14739 catalog_manager.cc:5484] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } }
I20240501 14:11:26.443789 15999 raft_consensus.cc:482] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:26.443857 15999 raft_consensus.cc:504] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:26.444145 15999 leader_election.cc:288] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411), 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130:40469)
I20240501 14:11:26.444418 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a99203e550254896a5e269c91201fa85" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05" is_pre_election: true
I20240501 14:11:26.444458 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a99203e550254896a5e269c91201fa85" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f49c2edf1b54d83811340aaea48e682" is_pre_election: true
I20240501 14:11:26.444527 14884 raft_consensus.cc:2454] T a99203e550254896a5e269c91201fa85 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in term 0.
I20240501 14:11:26.444542 15016 raft_consensus.cc:2454] T a99203e550254896a5e269c91201fa85 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in term 0.
I20240501 14:11:26.444773 15212 leader_election.cc:302] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f24c1c43aa2456eb67b2cbb9ed1a948, 5f49c2edf1b54d83811340aaea48e682; no voters:
I20240501 14:11:26.444940 15999 raft_consensus.cc:2781] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:11:26.444993 15999 raft_consensus.cc:482] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:11:26.445020 15999 raft_consensus.cc:3037] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:26.445662 15999 raft_consensus.cc:504] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:26.445955 15999 leader_election.cc:288] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 election: Requested vote from peers 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411), 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130:40469)
I20240501 14:11:26.446221 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a99203e550254896a5e269c91201fa85" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05"
I20240501 14:11:26.446261 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a99203e550254896a5e269c91201fa85" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f49c2edf1b54d83811340aaea48e682"
I20240501 14:11:26.446324 14884 raft_consensus.cc:3037] T a99203e550254896a5e269c91201fa85 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:26.446344 15016 raft_consensus.cc:3037] T a99203e550254896a5e269c91201fa85 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:26.447139 14884 raft_consensus.cc:2454] T a99203e550254896a5e269c91201fa85 P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in term 1.
I20240501 14:11:26.447139 15016 raft_consensus.cc:2454] T a99203e550254896a5e269c91201fa85 P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in term 1.
I20240501 14:11:26.447381 15214 leader_election.cc:302] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00035488ce0740898f6f92a508b01a05, 5f24c1c43aa2456eb67b2cbb9ed1a948; no voters:
I20240501 14:11:26.447520 15999 raft_consensus.cc:2781] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:11:26.447597 15999 raft_consensus.cc:686] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 LEADER]: Becoming Leader. State: Replica: 5f24c1c43aa2456eb67b2cbb9ed1a948, State: Running, Role: LEADER
I20240501 14:11:26.447721 15999 consensus_queue.cc:237] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:26.449050 14739 catalog_manager.cc:5484] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } }
I20240501 14:11:26.665194 15878 raft_consensus.cc:482] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:26.665330 15878 raft_consensus.cc:504] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } }
I20240501 14:11:26.666414 15878 leader_election.cc:288] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131:39965), 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132:40939)
I20240501 14:11:26.669914 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9713eefbff264eb592a424e4fbc5000c" candidate_uuid: "00035488ce0740898f6f92a508b01a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" is_pre_election: true
I20240501 14:11:26.669926 15279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9713eefbff264eb592a424e4fbc5000c" candidate_uuid: "00035488ce0740898f6f92a508b01a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" is_pre_election: true
I20240501 14:11:26.670051 15279 raft_consensus.cc:2454] T 9713eefbff264eb592a424e4fbc5000c P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00035488ce0740898f6f92a508b01a05 in term 0.
I20240501 14:11:26.670063 15147 raft_consensus.cc:2454] T 9713eefbff264eb592a424e4fbc5000c P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00035488ce0740898f6f92a508b01a05 in term 0.
I20240501 14:11:26.670418 14822 leader_election.cc:302] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00035488ce0740898f6f92a508b01a05, 5f24c1c43aa2456eb67b2cbb9ed1a948; no voters:
I20240501 14:11:26.670643 15878 raft_consensus.cc:2781] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:11:26.670701 15878 raft_consensus.cc:482] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:11:26.670737 15878 raft_consensus.cc:3037] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:26.671510 15878 raft_consensus.cc:504] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } }
I20240501 14:11:26.671849 15878 leader_election.cc:288] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05 [CANDIDATE]: Term 1 election: Requested vote from peers dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131:39965), 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132:40939)
I20240501 14:11:26.672161 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9713eefbff264eb592a424e4fbc5000c" candidate_uuid: "00035488ce0740898f6f92a508b01a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc6733e9b3741578b9bc25604dd3bc6"
I20240501 14:11:26.672178 15279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9713eefbff264eb592a424e4fbc5000c" candidate_uuid: "00035488ce0740898f6f92a508b01a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948"
I20240501 14:11:26.672283 15147 raft_consensus.cc:3037] T 9713eefbff264eb592a424e4fbc5000c P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:26.672312 15279 raft_consensus.cc:3037] T 9713eefbff264eb592a424e4fbc5000c P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:26.673032 15147 raft_consensus.cc:2454] T 9713eefbff264eb592a424e4fbc5000c P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00035488ce0740898f6f92a508b01a05 in term 1.
I20240501 14:11:26.673110 15279 raft_consensus.cc:2454] T 9713eefbff264eb592a424e4fbc5000c P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00035488ce0740898f6f92a508b01a05 in term 1.
I20240501 14:11:26.673332 14821 leader_election.cc:302] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00035488ce0740898f6f92a508b01a05, dcc6733e9b3741578b9bc25604dd3bc6; no voters:
I20240501 14:11:26.673503 15878 raft_consensus.cc:2781] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:11:26.673645 15878 raft_consensus.cc:686] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05 [term 1 LEADER]: Becoming Leader. State: Replica: 00035488ce0740898f6f92a508b01a05, State: Running, Role: LEADER
I20240501 14:11:26.673923 15878 consensus_queue.cc:237] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } }
I20240501 14:11:26.675838 14739 catalog_manager.cc:5484] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00035488ce0740898f6f92a508b01a05 (127.22.252.129). New cstate: current_term: 1 leader_uuid: "00035488ce0740898f6f92a508b01a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } }
W20240501 14:11:26.782506 16022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:11:26.782743 16022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:11:26.785841 16022 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 14:11:26.863270 15016 raft_consensus.cc:1261] T bae4a31367f746bba17d2540cd36c4d5 P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Refusing update from remote peer 5f24c1c43aa2456eb67b2cbb9ed1a948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:26.863415 14884 raft_consensus.cc:1261] T bae4a31367f746bba17d2540cd36c4d5 P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Refusing update from remote peer 5f24c1c43aa2456eb67b2cbb9ed1a948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:26.863767 15017 raft_consensus.cc:1261] T a99203e550254896a5e269c91201fa85 P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Refusing update from remote peer 5f24c1c43aa2456eb67b2cbb9ed1a948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:26.863879 15879 consensus_queue.cc:1035] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:26.864040 15999 consensus_queue.cc:1035] T bae4a31367f746bba17d2540cd36c4d5 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:26.864351 15999 consensus_queue.cc:1035] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:26.866636 14886 raft_consensus.cc:1261] T a99203e550254896a5e269c91201fa85 P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Refusing update from remote peer 5f24c1c43aa2456eb67b2cbb9ed1a948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:26.867007 15147 raft_consensus.cc:1261] T 9713eefbff264eb592a424e4fbc5000c P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Refusing update from remote peer 00035488ce0740898f6f92a508b01a05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:26.867723 15279 raft_consensus.cc:1261] T 9713eefbff264eb592a424e4fbc5000c P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Refusing update from remote peer 00035488ce0740898f6f92a508b01a05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:26.867993 15878 consensus_queue.cc:1035] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:26.871201 15878 consensus_queue.cc:1035] T 9713eefbff264eb592a424e4fbc5000c P 00035488ce0740898f6f92a508b01a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:26.874478 16054 consensus_queue.cc:1035] T a99203e550254896a5e269c91201fa85 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
INSERT report
rows total: 2000
time total: 97.1322 ms
time per row: 0.0485661 ms
I20240501 14:11:26.924873 14737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41026:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:11:26.929951 14995 tablet_service.cc:1467] Processing CreateTablet for tablet 5e181d6cd13b4ded8558d823047383cb (DEFAULT_TABLE table=rebalance_test_table_3 [id=72766721cfa44551814f01f131d4a79b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:26.930287 14995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e181d6cd13b4ded8558d823047383cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:26.930593 15127 tablet_service.cc:1467] Processing CreateTablet for tablet 5e181d6cd13b4ded8558d823047383cb (DEFAULT_TABLE table=rebalance_test_table_3 [id=72766721cfa44551814f01f131d4a79b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:26.930717 14865 tablet_service.cc:1467] Processing CreateTablet for tablet 5e181d6cd13b4ded8558d823047383cb (DEFAULT_TABLE table=rebalance_test_table_3 [id=72766721cfa44551814f01f131d4a79b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:26.930814 15127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e181d6cd13b4ded8558d823047383cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:26.930945 14865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e181d6cd13b4ded8558d823047383cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:26.932444 15126 tablet_service.cc:1467] Processing CreateTablet for tablet f4603185a78249388dbf10535eb11327 (DEFAULT_TABLE table=rebalance_test_table_3 [id=72766721cfa44551814f01f131d4a79b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:26.932653 15126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4603185a78249388dbf10535eb11327. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:26.932945 16075 tablet_bootstrap.cc:492] T 5e181d6cd13b4ded8558d823047383cb P 00035488ce0740898f6f92a508b01a05: Bootstrap starting.
I20240501 14:11:26.933101 14866 tablet_service.cc:1467] Processing CreateTablet for tablet a4f672c0b1b5454ea300f0afe8fcbd9e (DEFAULT_TABLE table=rebalance_test_table_3 [id=72766721cfa44551814f01f131d4a79b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:26.933295 14866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f672c0b1b5454ea300f0afe8fcbd9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:26.934103 16075 tablet_bootstrap.cc:654] T 5e181d6cd13b4ded8558d823047383cb P 00035488ce0740898f6f92a508b01a05: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:26.935108 16075 tablet_bootstrap.cc:492] T 5e181d6cd13b4ded8558d823047383cb P 00035488ce0740898f6f92a508b01a05: No bootstrap required, opened a new log
I20240501 14:11:26.935232 16075 ts_tablet_manager.cc:1367] T 5e181d6cd13b4ded8558d823047383cb P 00035488ce0740898f6f92a508b01a05: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:11:26.935330 15259 tablet_service.cc:1467] Processing CreateTablet for tablet a4f672c0b1b5454ea300f0afe8fcbd9e (DEFAULT_TABLE table=rebalance_test_table_3 [id=72766721cfa44551814f01f131d4a79b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:26.935376 15128 tablet_service.cc:1467] Processing CreateTablet for tablet a4f672c0b1b5454ea300f0afe8fcbd9e (DEFAULT_TABLE table=rebalance_test_table_3 [id=72766721cfa44551814f01f131d4a79b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:26.935547 15259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f672c0b1b5454ea300f0afe8fcbd9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:26.935578 15128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f672c0b1b5454ea300f0afe8fcbd9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:26.935850 16075 raft_consensus.cc:348] T 5e181d6cd13b4ded8558d823047383cb P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:26.936010 16075 raft_consensus.cc:374] T 5e181d6cd13b4ded8558d823047383cb P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:26.936077 16075 raft_consensus.cc:729] T 5e181d6cd13b4ded8558d823047383cb P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00035488ce0740898f6f92a508b01a05, State: Initialized, Role: FOLLOWER
I20240501 14:11:26.936220 16075 consensus_queue.cc:260] T 5e181d6cd13b4ded8558d823047383cb P 00035488ce0740898f6f92a508b01a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:26.936548 15258 tablet_service.cc:1467] Processing CreateTablet for tablet f4603185a78249388dbf10535eb11327 (DEFAULT_TABLE table=rebalance_test_table_3 [id=72766721cfa44551814f01f131d4a79b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:26.936738 15258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4603185a78249388dbf10535eb11327. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:26.937109 16075 ts_tablet_manager.cc:1398] T 5e181d6cd13b4ded8558d823047383cb P 00035488ce0740898f6f92a508b01a05: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:11:26.937315 16075 tablet_bootstrap.cc:492] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05: Bootstrap starting.
I20240501 14:11:26.938064 14995 tablet_service.cc:1467] Processing CreateTablet for tablet f4603185a78249388dbf10535eb11327 (DEFAULT_TABLE table=rebalance_test_table_3 [id=72766721cfa44551814f01f131d4a79b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:26.938297 14995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4603185a78249388dbf10535eb11327. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:26.939834 16078 tablet_bootstrap.cc:492] T f4603185a78249388dbf10535eb11327 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Bootstrap starting.
I20240501 14:11:26.940574 16078 tablet_bootstrap.cc:654] T f4603185a78249388dbf10535eb11327 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:26.940579 16075 tablet_bootstrap.cc:654] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:26.941963 16075 tablet_bootstrap.cc:492] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05: No bootstrap required, opened a new log
I20240501 14:11:26.942073 16075 ts_tablet_manager.cc:1367] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:11:26.942474 16075 raft_consensus.cc:348] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:26.942626 16075 raft_consensus.cc:374] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:26.942704 16075 raft_consensus.cc:729] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00035488ce0740898f6f92a508b01a05, State: Initialized, Role: FOLLOWER
I20240501 14:11:26.942873 16075 consensus_queue.cc:260] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:26.943238 16075 ts_tablet_manager.cc:1398] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:11:26.944193 16076 tablet_bootstrap.cc:492] T f4603185a78249388dbf10535eb11327 P dcc6733e9b3741578b9bc25604dd3bc6: Bootstrap starting.
I20240501 14:11:26.944227 16078 tablet_bootstrap.cc:492] T f4603185a78249388dbf10535eb11327 P 5f24c1c43aa2456eb67b2cbb9ed1a948: No bootstrap required, opened a new log
I20240501 14:11:26.944319 16078 ts_tablet_manager.cc:1367] T f4603185a78249388dbf10535eb11327 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:11:26.944598 16074 tablet_bootstrap.cc:492] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682: Bootstrap starting.
I20240501 14:11:26.944700 16078 raft_consensus.cc:348] T f4603185a78249388dbf10535eb11327 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:26.944782 16078 raft_consensus.cc:374] T f4603185a78249388dbf10535eb11327 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:26.944805 16078 raft_consensus.cc:729] T f4603185a78249388dbf10535eb11327 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f24c1c43aa2456eb67b2cbb9ed1a948, State: Initialized, Role: FOLLOWER
I20240501 14:11:26.944936 16078 consensus_queue.cc:260] T f4603185a78249388dbf10535eb11327 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:26.945166 16078 ts_tablet_manager.cc:1398] T f4603185a78249388dbf10535eb11327 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:26.945288 16078 tablet_bootstrap.cc:492] T a4f672c0b1b5454ea300f0afe8fcbd9e P 5f24c1c43aa2456eb67b2cbb9ed1a948: Bootstrap starting.
I20240501 14:11:26.946292 16078 tablet_bootstrap.cc:654] T a4f672c0b1b5454ea300f0afe8fcbd9e P 5f24c1c43aa2456eb67b2cbb9ed1a948: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:26.947185 16076 tablet_bootstrap.cc:654] T f4603185a78249388dbf10535eb11327 P dcc6733e9b3741578b9bc25604dd3bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:26.947546 16078 tablet_bootstrap.cc:492] T a4f672c0b1b5454ea300f0afe8fcbd9e P 5f24c1c43aa2456eb67b2cbb9ed1a948: No bootstrap required, opened a new log
I20240501 14:11:26.947649 16078 ts_tablet_manager.cc:1367] T a4f672c0b1b5454ea300f0afe8fcbd9e P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:11:26.948086 16074 tablet_bootstrap.cc:654] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:26.948045 16078 raft_consensus.cc:348] T a4f672c0b1b5454ea300f0afe8fcbd9e P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:26.948125 16078 raft_consensus.cc:374] T a4f672c0b1b5454ea300f0afe8fcbd9e P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:26.948153 16078 raft_consensus.cc:729] T a4f672c0b1b5454ea300f0afe8fcbd9e P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f24c1c43aa2456eb67b2cbb9ed1a948, State: Initialized, Role: FOLLOWER
I20240501 14:11:26.948314 16078 consensus_queue.cc:260] T a4f672c0b1b5454ea300f0afe8fcbd9e P 5f24c1c43aa2456eb67b2cbb9ed1a948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:26.948545 16078 ts_tablet_manager.cc:1398] T a4f672c0b1b5454ea300f0afe8fcbd9e P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:26.948725 16076 tablet_bootstrap.cc:492] T f4603185a78249388dbf10535eb11327 P dcc6733e9b3741578b9bc25604dd3bc6: No bootstrap required, opened a new log
I20240501 14:11:26.948849 16076 ts_tablet_manager.cc:1367] T f4603185a78249388dbf10535eb11327 P dcc6733e9b3741578b9bc25604dd3bc6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:11:26.949388 16076 raft_consensus.cc:348] T f4603185a78249388dbf10535eb11327 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:26.949539 16076 raft_consensus.cc:374] T f4603185a78249388dbf10535eb11327 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:26.949615 16076 raft_consensus.cc:729] T f4603185a78249388dbf10535eb11327 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc6733e9b3741578b9bc25604dd3bc6, State: Initialized, Role: FOLLOWER
I20240501 14:11:26.949622 16074 tablet_bootstrap.cc:492] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682: No bootstrap required, opened a new log
I20240501 14:11:26.949709 16074 ts_tablet_manager.cc:1367] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:11:26.949851 16076 consensus_queue.cc:260] T f4603185a78249388dbf10535eb11327 P dcc6733e9b3741578b9bc25604dd3bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:26.950114 16074 raft_consensus.cc:348] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:26.950230 16074 raft_consensus.cc:374] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:26.950266 16074 raft_consensus.cc:729] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f49c2edf1b54d83811340aaea48e682, State: Initialized, Role: FOLLOWER
I20240501 14:11:26.950414 16076 ts_tablet_manager.cc:1398] T f4603185a78249388dbf10535eb11327 P dcc6733e9b3741578b9bc25604dd3bc6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:11:26.950389 16074 consensus_queue.cc:260] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:26.950646 16074 ts_tablet_manager.cc:1398] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:26.950784 16076 tablet_bootstrap.cc:492] T 5e181d6cd13b4ded8558d823047383cb P dcc6733e9b3741578b9bc25604dd3bc6: Bootstrap starting.
I20240501 14:11:26.951134 16074 tablet_bootstrap.cc:492] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682: Bootstrap starting.
I20240501 14:11:26.951568 16076 tablet_bootstrap.cc:654] T 5e181d6cd13b4ded8558d823047383cb P dcc6733e9b3741578b9bc25604dd3bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:26.951953 16074 tablet_bootstrap.cc:654] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:26.952585 16076 tablet_bootstrap.cc:492] T 5e181d6cd13b4ded8558d823047383cb P dcc6733e9b3741578b9bc25604dd3bc6: No bootstrap required, opened a new log
I20240501 14:11:26.952675 16076 ts_tablet_manager.cc:1367] T 5e181d6cd13b4ded8558d823047383cb P dcc6733e9b3741578b9bc25604dd3bc6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:11:26.952991 16074 tablet_bootstrap.cc:492] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682: No bootstrap required, opened a new log
I20240501 14:11:26.953066 16074 ts_tablet_manager.cc:1367] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:11:26.953122 16076 raft_consensus.cc:348] T 5e181d6cd13b4ded8558d823047383cb P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:26.953208 16076 raft_consensus.cc:374] T 5e181d6cd13b4ded8558d823047383cb P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:26.953235 16076 raft_consensus.cc:729] T 5e181d6cd13b4ded8558d823047383cb P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc6733e9b3741578b9bc25604dd3bc6, State: Initialized, Role: FOLLOWER
I20240501 14:11:26.953349 16076 consensus_queue.cc:260] T 5e181d6cd13b4ded8558d823047383cb P dcc6733e9b3741578b9bc25604dd3bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:26.953595 16076 ts_tablet_manager.cc:1398] T 5e181d6cd13b4ded8558d823047383cb P dcc6733e9b3741578b9bc25604dd3bc6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:26.953771 16076 tablet_bootstrap.cc:492] T a4f672c0b1b5454ea300f0afe8fcbd9e P dcc6733e9b3741578b9bc25604dd3bc6: Bootstrap starting.
I20240501 14:11:26.953980 16074 raft_consensus.cc:348] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:26.954074 16074 raft_consensus.cc:374] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:26.954104 16074 raft_consensus.cc:729] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f49c2edf1b54d83811340aaea48e682, State: Initialized, Role: FOLLOWER
I20240501 14:11:26.954243 16074 consensus_queue.cc:260] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:26.954464 16074 ts_tablet_manager.cc:1398] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:11:26.954617 16076 tablet_bootstrap.cc:654] T a4f672c0b1b5454ea300f0afe8fcbd9e P dcc6733e9b3741578b9bc25604dd3bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:26.955497 16076 tablet_bootstrap.cc:492] T a4f672c0b1b5454ea300f0afe8fcbd9e P dcc6733e9b3741578b9bc25604dd3bc6: No bootstrap required, opened a new log
I20240501 14:11:26.955574 16076 ts_tablet_manager.cc:1367] T a4f672c0b1b5454ea300f0afe8fcbd9e P dcc6733e9b3741578b9bc25604dd3bc6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:11:26.955883 16076 raft_consensus.cc:348] T a4f672c0b1b5454ea300f0afe8fcbd9e P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:26.955974 16076 raft_consensus.cc:374] T a4f672c0b1b5454ea300f0afe8fcbd9e P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:26.956009 16076 raft_consensus.cc:729] T a4f672c0b1b5454ea300f0afe8fcbd9e P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc6733e9b3741578b9bc25604dd3bc6, State: Initialized, Role: FOLLOWER
I20240501 14:11:26.956126 16076 consensus_queue.cc:260] T a4f672c0b1b5454ea300f0afe8fcbd9e P dcc6733e9b3741578b9bc25604dd3bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:26.956383 16076 ts_tablet_manager.cc:1398] T a4f672c0b1b5454ea300f0afe8fcbd9e P dcc6733e9b3741578b9bc25604dd3bc6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:26.959450 15967 raft_consensus.cc:482] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:26.959524 15967 raft_consensus.cc:504] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:26.959875 15967 leader_election.cc:288] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411), dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131:39965)
I20240501 14:11:26.960520 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e181d6cd13b4ded8558d823047383cb" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" is_pre_election: true
I20240501 14:11:26.960659 15147 raft_consensus.cc:2454] T 5e181d6cd13b4ded8558d823047383cb P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 0.
I20240501 14:11:26.960922 14952 leader_election.cc:302] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f49c2edf1b54d83811340aaea48e682, dcc6733e9b3741578b9bc25604dd3bc6; no voters:
I20240501 14:11:26.961097 15967 raft_consensus.cc:2781] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:11:26.961143 15967 raft_consensus.cc:482] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:11:26.961165 15967 raft_consensus.cc:3037] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:26.961818 15967 raft_consensus.cc:504] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:26.962111 15967 leader_election.cc:288] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 election: Requested vote from peers 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411), dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131:39965)
I20240501 14:11:26.962370 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e181d6cd13b4ded8558d823047383cb" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc6733e9b3741578b9bc25604dd3bc6"
I20240501 14:11:26.962466 15147 raft_consensus.cc:3037] T 5e181d6cd13b4ded8558d823047383cb P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:26.963136 15147 raft_consensus.cc:2454] T 5e181d6cd13b4ded8558d823047383cb P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 1.
I20240501 14:11:26.963356 14952 leader_election.cc:302] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f49c2edf1b54d83811340aaea48e682, dcc6733e9b3741578b9bc25604dd3bc6; no voters:
I20240501 14:11:26.963346 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e181d6cd13b4ded8558d823047383cb" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05" is_pre_election: true
I20240501 14:11:26.963399 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e181d6cd13b4ded8558d823047383cb" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05"
I20240501 14:11:26.963450 14884 raft_consensus.cc:2454] T 5e181d6cd13b4ded8558d823047383cb P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 0.
I20240501 14:11:26.963503 15967 raft_consensus.cc:2781] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:11:26.963644 15967 raft_consensus.cc:686] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682 [term 1 LEADER]: Becoming Leader. State: Replica: 5f49c2edf1b54d83811340aaea48e682, State: Running, Role: LEADER
I20240501 14:11:26.963817 15967 consensus_queue.cc:237] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:26.965191 14744 catalog_manager.cc:5484] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "5f49c2edf1b54d83811340aaea48e682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } }
I20240501 14:11:26.975450 15967 raft_consensus.cc:482] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:26.975522 15967 raft_consensus.cc:504] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:26.975747 15967 leader_election.cc:288] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131:39965), 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132:40939)
I20240501 14:11:26.976014 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4603185a78249388dbf10535eb11327" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" is_pre_election: true
I20240501 14:11:26.976102 15147 raft_consensus.cc:2454] T f4603185a78249388dbf10535eb11327 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 0.
I20240501 14:11:26.976091 15279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4603185a78249388dbf10535eb11327" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" is_pre_election: true
I20240501 14:11:26.976218 15279 raft_consensus.cc:2454] T f4603185a78249388dbf10535eb11327 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 0.
I20240501 14:11:26.976357 14952 leader_election.cc:302] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f49c2edf1b54d83811340aaea48e682, dcc6733e9b3741578b9bc25604dd3bc6; no voters:
I20240501 14:11:26.976508 15967 raft_consensus.cc:2781] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:11:26.976557 15967 raft_consensus.cc:482] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:11:26.976581 15967 raft_consensus.cc:3037] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:26.977128 15967 raft_consensus.cc:504] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:26.977396 15967 leader_election.cc:288] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 election: Requested vote from peers dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131:39965), 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132:40939)
I20240501 14:11:26.977649 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4603185a78249388dbf10535eb11327" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc6733e9b3741578b9bc25604dd3bc6"
I20240501 14:11:26.977733 15279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4603185a78249388dbf10535eb11327" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948"
I20240501 14:11:26.977764 15147 raft_consensus.cc:3037] T f4603185a78249388dbf10535eb11327 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:26.977829 15279 raft_consensus.cc:3037] T f4603185a78249388dbf10535eb11327 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:26.978401 15147 raft_consensus.cc:2454] T f4603185a78249388dbf10535eb11327 P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 1.
I20240501 14:11:26.978547 15279 raft_consensus.cc:2454] T f4603185a78249388dbf10535eb11327 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 1.
I20240501 14:11:26.978665 14952 leader_election.cc:302] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f49c2edf1b54d83811340aaea48e682, dcc6733e9b3741578b9bc25604dd3bc6; no voters:
I20240501 14:11:26.978935 15967 raft_consensus.cc:2781] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:11:26.979030 15967 raft_consensus.cc:686] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682 [term 1 LEADER]: Becoming Leader. State: Replica: 5f49c2edf1b54d83811340aaea48e682, State: Running, Role: LEADER
I20240501 14:11:26.979151 15967 consensus_queue.cc:237] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:26.980505 14744 catalog_manager.cc:5484] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "5f49c2edf1b54d83811340aaea48e682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } }
I20240501 14:11:27.040544 15878 raft_consensus.cc:482] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:27.040668 15878 raft_consensus.cc:504] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:27.041211 15878 leader_election.cc:288] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132:40939), dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131:39965)
I20240501 14:11:27.041538 15279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f672c0b1b5454ea300f0afe8fcbd9e" candidate_uuid: "00035488ce0740898f6f92a508b01a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" is_pre_election: true
I20240501 14:11:27.041599 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f672c0b1b5454ea300f0afe8fcbd9e" candidate_uuid: "00035488ce0740898f6f92a508b01a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" is_pre_election: true
I20240501 14:11:27.041726 15147 raft_consensus.cc:2454] T a4f672c0b1b5454ea300f0afe8fcbd9e P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00035488ce0740898f6f92a508b01a05 in term 0.
I20240501 14:11:27.041728 15279 raft_consensus.cc:2454] T a4f672c0b1b5454ea300f0afe8fcbd9e P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00035488ce0740898f6f92a508b01a05 in term 0.
I20240501 14:11:27.041975 14822 leader_election.cc:302] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00035488ce0740898f6f92a508b01a05, 5f24c1c43aa2456eb67b2cbb9ed1a948; no voters:
I20240501 14:11:27.042133 15878 raft_consensus.cc:2781] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:11:27.042193 15878 raft_consensus.cc:482] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:11:27.042234 15878 raft_consensus.cc:3037] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:27.043026 15878 raft_consensus.cc:504] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:27.043303 15878 leader_election.cc:288] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05 [CANDIDATE]: Term 1 election: Requested vote from peers 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132:40939), dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131:39965)
I20240501 14:11:27.043607 15147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f672c0b1b5454ea300f0afe8fcbd9e" candidate_uuid: "00035488ce0740898f6f92a508b01a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc6733e9b3741578b9bc25604dd3bc6"
I20240501 14:11:27.043625 15279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f672c0b1b5454ea300f0afe8fcbd9e" candidate_uuid: "00035488ce0740898f6f92a508b01a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948"
I20240501 14:11:27.043712 15279 raft_consensus.cc:3037] T a4f672c0b1b5454ea300f0afe8fcbd9e P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:27.043712 15147 raft_consensus.cc:3037] T a4f672c0b1b5454ea300f0afe8fcbd9e P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:27.044448 15279 raft_consensus.cc:2454] T a4f672c0b1b5454ea300f0afe8fcbd9e P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00035488ce0740898f6f92a508b01a05 in term 1.
I20240501 14:11:27.044548 15147 raft_consensus.cc:2454] T a4f672c0b1b5454ea300f0afe8fcbd9e P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00035488ce0740898f6f92a508b01a05 in term 1.
I20240501 14:11:27.044705 14822 leader_election.cc:302] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00035488ce0740898f6f92a508b01a05, 5f24c1c43aa2456eb67b2cbb9ed1a948; no voters:
I20240501 14:11:27.044860 15878 raft_consensus.cc:2781] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:11:27.044929 15878 raft_consensus.cc:686] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05 [term 1 LEADER]: Becoming Leader. State: Replica: 00035488ce0740898f6f92a508b01a05, State: Running, Role: LEADER
I20240501 14:11:27.045058 15878 consensus_queue.cc:237] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:27.046447 14744 catalog_manager.cc:5484] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00035488ce0740898f6f92a508b01a05 (127.22.252.129). New cstate: current_term: 1 leader_uuid: "00035488ce0740898f6f92a508b01a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } }
W20240501 14:11:27.158246 16092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:11:27.158442 16092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:11:27.161443 16092 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 14:11:27.249702 15279 raft_consensus.cc:1261] T a4f672c0b1b5454ea300f0afe8fcbd9e P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Refusing update from remote peer 00035488ce0740898f6f92a508b01a05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:27.249702 15147 raft_consensus.cc:1261] T a4f672c0b1b5454ea300f0afe8fcbd9e P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Refusing update from remote peer 00035488ce0740898f6f92a508b01a05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:27.250147 15878 consensus_queue.cc:1035] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:27.250473 15878 consensus_queue.cc:1035] T a4f672c0b1b5454ea300f0afe8fcbd9e P 00035488ce0740898f6f92a508b01a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:27.250666 15147 raft_consensus.cc:1261] T 5e181d6cd13b4ded8558d823047383cb P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Refusing update from remote peer 5f49c2edf1b54d83811340aaea48e682: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:27.252099 16088 consensus_queue.cc:1035] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:27.253295 15279 raft_consensus.cc:1261] T f4603185a78249388dbf10535eb11327 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Refusing update from remote peer 5f49c2edf1b54d83811340aaea48e682: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:27.253782 16088 consensus_queue.cc:1035] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:27.256716 15146 raft_consensus.cc:1261] T f4603185a78249388dbf10535eb11327 P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Refusing update from remote peer 5f49c2edf1b54d83811340aaea48e682: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:27.256943 14886 raft_consensus.cc:3037] T 5e181d6cd13b4ded8558d823047383cb P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:27.258078 14886 raft_consensus.cc:1261] T 5e181d6cd13b4ded8558d823047383cb P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Refusing update from remote peer 5f49c2edf1b54d83811340aaea48e682: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:27.258272 15967 consensus_queue.cc:1035] T f4603185a78249388dbf10535eb11327 P 5f49c2edf1b54d83811340aaea48e682 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:27.265370 15967 consensus_queue.cc:1035] T 5e181d6cd13b4ded8558d823047383cb P 5f49c2edf1b54d83811340aaea48e682 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 109.238 ms
time per row: 0.0546188 ms
I20240501 14:11:27.310119 14744 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41026:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:11:27.314854 14866 tablet_service.cc:1467] Processing CreateTablet for tablet 45d37ef41bfb48cea9dcc05c45f782df (DEFAULT_TABLE table=rebalance_test_table_4 [id=c8f6808198f94d0fb04aafd29fd29d38]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:27.314895 14995 tablet_service.cc:1467] Processing CreateTablet for tablet 45d37ef41bfb48cea9dcc05c45f782df (DEFAULT_TABLE table=rebalance_test_table_4 [id=c8f6808198f94d0fb04aafd29fd29d38]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:27.315054 15259 tablet_service.cc:1467] Processing CreateTablet for tablet d2405c8c944547f783912779f5c4d838 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c8f6808198f94d0fb04aafd29fd29d38]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:27.315165 14995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d37ef41bfb48cea9dcc05c45f782df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:27.315165 14866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d37ef41bfb48cea9dcc05c45f782df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:27.315359 15259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2405c8c944547f783912779f5c4d838. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:27.316872 15258 tablet_service.cc:1467] Processing CreateTablet for tablet 04aa763389aa4855babf3f9035177734 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c8f6808198f94d0fb04aafd29fd29d38]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:27.317062 15258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04aa763389aa4855babf3f9035177734. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:27.317183 16074 tablet_bootstrap.cc:492] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682: Bootstrap starting.
I20240501 14:11:27.317553 16075 tablet_bootstrap.cc:492] T 45d37ef41bfb48cea9dcc05c45f782df P 00035488ce0740898f6f92a508b01a05: Bootstrap starting.
I20240501 14:11:27.317564 16078 tablet_bootstrap.cc:492] T d2405c8c944547f783912779f5c4d838 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Bootstrap starting.
I20240501 14:11:27.318241 16074 tablet_bootstrap.cc:654] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:27.318372 16075 tablet_bootstrap.cc:654] T 45d37ef41bfb48cea9dcc05c45f782df P 00035488ce0740898f6f92a508b01a05: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:27.318434 16078 tablet_bootstrap.cc:654] T d2405c8c944547f783912779f5c4d838 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:27.319231 15127 tablet_service.cc:1467] Processing CreateTablet for tablet 04aa763389aa4855babf3f9035177734 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c8f6808198f94d0fb04aafd29fd29d38]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:27.319489 15127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04aa763389aa4855babf3f9035177734. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:27.319442 14865 tablet_service.cc:1467] Processing CreateTablet for tablet 04aa763389aa4855babf3f9035177734 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c8f6808198f94d0fb04aafd29fd29d38]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:27.319555 16075 tablet_bootstrap.cc:492] T 45d37ef41bfb48cea9dcc05c45f782df P 00035488ce0740898f6f92a508b01a05: No bootstrap required, opened a new log
I20240501 14:11:27.319624 16075 ts_tablet_manager.cc:1367] T 45d37ef41bfb48cea9dcc05c45f782df P 00035488ce0740898f6f92a508b01a05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:11:27.319648 14865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04aa763389aa4855babf3f9035177734. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:27.319808 16078 tablet_bootstrap.cc:492] T d2405c8c944547f783912779f5c4d838 P 5f24c1c43aa2456eb67b2cbb9ed1a948: No bootstrap required, opened a new log
I20240501 14:11:27.319873 16078 ts_tablet_manager.cc:1367] T d2405c8c944547f783912779f5c4d838 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:11:27.320035 16075 raft_consensus.cc:348] T 45d37ef41bfb48cea9dcc05c45f782df P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:27.320145 16075 raft_consensus.cc:374] T 45d37ef41bfb48cea9dcc05c45f782df P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:27.320173 16075 raft_consensus.cc:729] T 45d37ef41bfb48cea9dcc05c45f782df P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00035488ce0740898f6f92a508b01a05, State: Initialized, Role: FOLLOWER
I20240501 14:11:27.320286 16078 raft_consensus.cc:348] T d2405c8c944547f783912779f5c4d838 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } }
I20240501 14:11:27.320322 16075 consensus_queue.cc:260] T 45d37ef41bfb48cea9dcc05c45f782df P 00035488ce0740898f6f92a508b01a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:27.320387 16078 raft_consensus.cc:374] T d2405c8c944547f783912779f5c4d838 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:27.320418 16078 raft_consensus.cc:729] T d2405c8c944547f783912779f5c4d838 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f24c1c43aa2456eb67b2cbb9ed1a948, State: Initialized, Role: FOLLOWER
I20240501 14:11:27.320528 16078 consensus_queue.cc:260] T d2405c8c944547f783912779f5c4d838 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } }
I20240501 14:11:27.320611 16075 ts_tablet_manager.cc:1398] T 45d37ef41bfb48cea9dcc05c45f782df P 00035488ce0740898f6f92a508b01a05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:27.320819 16078 ts_tablet_manager.cc:1398] T d2405c8c944547f783912779f5c4d838 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:27.320974 16078 tablet_bootstrap.cc:492] T 04aa763389aa4855babf3f9035177734 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Bootstrap starting.
I20240501 14:11:27.317370 15128 tablet_service.cc:1467] Processing CreateTablet for tablet 45d37ef41bfb48cea9dcc05c45f782df (DEFAULT_TABLE table=rebalance_test_table_4 [id=c8f6808198f94d0fb04aafd29fd29d38]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:27.321543 16076 tablet_bootstrap.cc:492] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6: Bootstrap starting.
I20240501 14:11:27.321646 15128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d37ef41bfb48cea9dcc05c45f782df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:27.321815 16078 tablet_bootstrap.cc:654] T 04aa763389aa4855babf3f9035177734 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:27.322410 16076 tablet_bootstrap.cc:654] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:27.322688 16078 tablet_bootstrap.cc:492] T 04aa763389aa4855babf3f9035177734 P 5f24c1c43aa2456eb67b2cbb9ed1a948: No bootstrap required, opened a new log
I20240501 14:11:27.322763 16078 ts_tablet_manager.cc:1367] T 04aa763389aa4855babf3f9035177734 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:11:27.323207 16078 raft_consensus.cc:348] T 04aa763389aa4855babf3f9035177734 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:27.323287 16078 raft_consensus.cc:374] T 04aa763389aa4855babf3f9035177734 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:27.323354 16078 raft_consensus.cc:729] T 04aa763389aa4855babf3f9035177734 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f24c1c43aa2456eb67b2cbb9ed1a948, State: Initialized, Role: FOLLOWER
I20240501 14:11:27.323464 16076 tablet_bootstrap.cc:492] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6: No bootstrap required, opened a new log
I20240501 14:11:27.323488 16078 consensus_queue.cc:260] T 04aa763389aa4855babf3f9035177734 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:27.323554 16076 ts_tablet_manager.cc:1367] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:11:27.323804 16078 ts_tablet_manager.cc:1398] T 04aa763389aa4855babf3f9035177734 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:27.324003 16076 raft_consensus.cc:348] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:27.324131 16076 raft_consensus.cc:374] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:27.324172 16076 raft_consensus.cc:729] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc6733e9b3741578b9bc25604dd3bc6, State: Initialized, Role: FOLLOWER
I20240501 14:11:27.324311 16076 consensus_queue.cc:260] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:27.324487 16054 raft_consensus.cc:482] T d2405c8c944547f783912779f5c4d838 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:27.324656 16076 ts_tablet_manager.cc:1398] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:27.324812 16076 tablet_bootstrap.cc:492] T 45d37ef41bfb48cea9dcc05c45f782df P dcc6733e9b3741578b9bc25604dd3bc6: Bootstrap starting.
I20240501 14:11:27.324556 16054 raft_consensus.cc:504] T d2405c8c944547f783912779f5c4d838 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } }
I20240501 14:11:27.325315 14995 tablet_service.cc:1467] Processing CreateTablet for tablet d2405c8c944547f783912779f5c4d838 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c8f6808198f94d0fb04aafd29fd29d38]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:27.325371 16054 leader_election.cc:288] T d2405c8c944547f783912779f5c4d838 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130:40469), 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411)
I20240501 14:11:27.325338 14866 tablet_service.cc:1467] Processing CreateTablet for tablet d2405c8c944547f783912779f5c4d838 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c8f6808198f94d0fb04aafd29fd29d38]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:27.325563 14995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2405c8c944547f783912779f5c4d838. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:27.325564 14866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2405c8c944547f783912779f5c4d838. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:11:27.326326 15214 leader_election.cc:341] T d2405c8c944547f783912779f5c4d838 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411): Not found: Tablet not found: d2405c8c944547f783912779f5c4d838
I20240501 14:11:27.327414 16075 tablet_bootstrap.cc:492] T 04aa763389aa4855babf3f9035177734 P 00035488ce0740898f6f92a508b01a05: Bootstrap starting.
I20240501 14:11:27.328109 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2405c8c944547f783912779f5c4d838" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f49c2edf1b54d83811340aaea48e682" is_pre_election: true
I20240501 14:11:27.328543 16075 tablet_bootstrap.cc:654] T 04aa763389aa4855babf3f9035177734 P 00035488ce0740898f6f92a508b01a05: Neither blocks nor log segments found. Creating new log.
W20240501 14:11:27.328610 15212 leader_election.cc:341] T d2405c8c944547f783912779f5c4d838 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130:40469): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:11:27.328675 15212 leader_election.cc:302] T d2405c8c944547f783912779f5c4d838 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f24c1c43aa2456eb67b2cbb9ed1a948; no voters: 00035488ce0740898f6f92a508b01a05, 5f49c2edf1b54d83811340aaea48e682
I20240501 14:11:27.329591 16054 raft_consensus.cc:2726] T d2405c8c944547f783912779f5c4d838 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:11:27.329686 16074 tablet_bootstrap.cc:492] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682: No bootstrap required, opened a new log
I20240501 14:11:27.329767 16075 tablet_bootstrap.cc:492] T 04aa763389aa4855babf3f9035177734 P 00035488ce0740898f6f92a508b01a05: No bootstrap required, opened a new log
I20240501 14:11:27.329782 16074 ts_tablet_manager.cc:1367] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20240501 14:11:27.329843 16075 ts_tablet_manager.cc:1367] T 04aa763389aa4855babf3f9035177734 P 00035488ce0740898f6f92a508b01a05: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:11:27.330217 16075 raft_consensus.cc:348] T 04aa763389aa4855babf3f9035177734 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:27.330216 16074 raft_consensus.cc:348] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:27.330319 16074 raft_consensus.cc:374] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:27.330319 16075 raft_consensus.cc:374] T 04aa763389aa4855babf3f9035177734 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:27.330353 16075 raft_consensus.cc:729] T 04aa763389aa4855babf3f9035177734 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00035488ce0740898f6f92a508b01a05, State: Initialized, Role: FOLLOWER
I20240501 14:11:27.330353 16074 raft_consensus.cc:729] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f49c2edf1b54d83811340aaea48e682, State: Initialized, Role: FOLLOWER
I20240501 14:11:27.330469 16074 consensus_queue.cc:260] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:27.330469 16075 consensus_queue.cc:260] T 04aa763389aa4855babf3f9035177734 P 00035488ce0740898f6f92a508b01a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:27.330718 16075 ts_tablet_manager.cc:1398] T 04aa763389aa4855babf3f9035177734 P 00035488ce0740898f6f92a508b01a05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:27.330767 16074 ts_tablet_manager.cc:1398] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 14:11:27.330878 16075 tablet_bootstrap.cc:492] T d2405c8c944547f783912779f5c4d838 P 00035488ce0740898f6f92a508b01a05: Bootstrap starting.
I20240501 14:11:27.330919 16074 tablet_bootstrap.cc:492] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682: Bootstrap starting.
I20240501 14:11:27.331758 16074 tablet_bootstrap.cc:654] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:27.331758 16075 tablet_bootstrap.cc:654] T d2405c8c944547f783912779f5c4d838 P 00035488ce0740898f6f92a508b01a05: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:27.332912 16074 tablet_bootstrap.cc:492] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682: No bootstrap required, opened a new log
I20240501 14:11:27.332994 16074 ts_tablet_manager.cc:1367] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:11:27.333390 16074 raft_consensus.cc:348] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } }
I20240501 14:11:27.333503 16074 raft_consensus.cc:374] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:27.333537 16074 raft_consensus.cc:729] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f49c2edf1b54d83811340aaea48e682, State: Initialized, Role: FOLLOWER
I20240501 14:11:27.333684 16076 tablet_bootstrap.cc:654] T 45d37ef41bfb48cea9dcc05c45f782df P dcc6733e9b3741578b9bc25604dd3bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:27.333691 16074 consensus_queue.cc:260] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } }
I20240501 14:11:27.333993 16074 ts_tablet_manager.cc:1398] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:11:27.335122 16076 tablet_bootstrap.cc:492] T 45d37ef41bfb48cea9dcc05c45f782df P dcc6733e9b3741578b9bc25604dd3bc6: No bootstrap required, opened a new log
I20240501 14:11:27.335151 16075 tablet_bootstrap.cc:492] T d2405c8c944547f783912779f5c4d838 P 00035488ce0740898f6f92a508b01a05: No bootstrap required, opened a new log
I20240501 14:11:27.335198 16076 ts_tablet_manager.cc:1367] T 45d37ef41bfb48cea9dcc05c45f782df P dcc6733e9b3741578b9bc25604dd3bc6: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:11:27.335206 16075 ts_tablet_manager.cc:1367] T d2405c8c944547f783912779f5c4d838 P 00035488ce0740898f6f92a508b01a05: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:11:27.335551 16075 raft_consensus.cc:348] T d2405c8c944547f783912779f5c4d838 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } }
I20240501 14:11:27.335551 16076 raft_consensus.cc:348] T 45d37ef41bfb48cea9dcc05c45f782df P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:27.335645 16075 raft_consensus.cc:374] T d2405c8c944547f783912779f5c4d838 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:27.335645 16076 raft_consensus.cc:374] T 45d37ef41bfb48cea9dcc05c45f782df P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:27.335685 16076 raft_consensus.cc:729] T 45d37ef41bfb48cea9dcc05c45f782df P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc6733e9b3741578b9bc25604dd3bc6, State: Initialized, Role: FOLLOWER
I20240501 14:11:27.335685 16075 raft_consensus.cc:729] T d2405c8c944547f783912779f5c4d838 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00035488ce0740898f6f92a508b01a05, State: Initialized, Role: FOLLOWER
I20240501 14:11:27.335831 16076 consensus_queue.cc:260] T 45d37ef41bfb48cea9dcc05c45f782df P dcc6733e9b3741578b9bc25604dd3bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:27.336112 16076 ts_tablet_manager.cc:1398] T 45d37ef41bfb48cea9dcc05c45f782df P dcc6733e9b3741578b9bc25604dd3bc6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:27.336339 16075 consensus_queue.cc:260] T d2405c8c944547f783912779f5c4d838 P 00035488ce0740898f6f92a508b01a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } }
I20240501 14:11:27.336619 16075 ts_tablet_manager.cc:1398] T d2405c8c944547f783912779f5c4d838 P 00035488ce0740898f6f92a508b01a05: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:11:27.366050 16128 raft_consensus.cc:482] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:27.366153 16128 raft_consensus.cc:504] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } }
I20240501 14:11:27.366535 16128 leader_election.cc:288] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132:40939), 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411)
I20240501 14:11:27.366792 15279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2405c8c944547f783912779f5c4d838" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" is_pre_election: true
I20240501 14:11:27.366886 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2405c8c944547f783912779f5c4d838" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05" is_pre_election: true
I20240501 14:11:27.366957 15279 raft_consensus.cc:2454] T d2405c8c944547f783912779f5c4d838 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 0.
I20240501 14:11:27.367033 14886 raft_consensus.cc:2454] T d2405c8c944547f783912779f5c4d838 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 0.
I20240501 14:11:27.367236 14953 leader_election.cc:302] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f24c1c43aa2456eb67b2cbb9ed1a948, 5f49c2edf1b54d83811340aaea48e682; no voters:
I20240501 14:11:27.367410 16128 raft_consensus.cc:2781] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:11:27.367476 16128 raft_consensus.cc:482] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:11:27.367516 16128 raft_consensus.cc:3037] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:27.368382 16128 raft_consensus.cc:504] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } }
I20240501 14:11:27.368661 16128 leader_election.cc:288] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 election: Requested vote from peers 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132:40939), 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411)
I20240501 14:11:27.368925 15279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2405c8c944547f783912779f5c4d838" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948"
I20240501 14:11:27.368953 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2405c8c944547f783912779f5c4d838" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05"
I20240501 14:11:27.369067 15279 raft_consensus.cc:3037] T d2405c8c944547f783912779f5c4d838 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:27.369066 14886 raft_consensus.cc:3037] T d2405c8c944547f783912779f5c4d838 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:27.369933 14886 raft_consensus.cc:2454] T d2405c8c944547f783912779f5c4d838 P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 1.
I20240501 14:11:27.369933 15279 raft_consensus.cc:2454] T d2405c8c944547f783912779f5c4d838 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 1.
I20240501 14:11:27.370199 14953 leader_election.cc:302] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f24c1c43aa2456eb67b2cbb9ed1a948, 5f49c2edf1b54d83811340aaea48e682; no voters:
I20240501 14:11:27.370411 16128 raft_consensus.cc:2781] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:11:27.370481 16128 raft_consensus.cc:686] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682 [term 1 LEADER]: Becoming Leader. State: Replica: 5f49c2edf1b54d83811340aaea48e682, State: Running, Role: LEADER
I20240501 14:11:27.370610 16128 consensus_queue.cc:237] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } }
I20240501 14:11:27.371910 14737 catalog_manager.cc:5484] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "5f49c2edf1b54d83811340aaea48e682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } }
I20240501 14:11:27.472105 16134 raft_consensus.cc:482] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:27.472247 16134 raft_consensus.cc:504] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:27.472680 16134 leader_election.cc:288] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411), 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132:40939)
I20240501 14:11:27.473031 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04aa763389aa4855babf3f9035177734" candidate_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05" is_pre_election: true
I20240501 14:11:27.473052 15279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04aa763389aa4855babf3f9035177734" candidate_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" is_pre_election: true
I20240501 14:11:27.473157 14886 raft_consensus.cc:2454] T 04aa763389aa4855babf3f9035177734 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc6733e9b3741578b9bc25604dd3bc6 in term 0.
I20240501 14:11:27.473196 15279 raft_consensus.cc:2454] T 04aa763389aa4855babf3f9035177734 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc6733e9b3741578b9bc25604dd3bc6 in term 0.
I20240501 14:11:27.473465 15084 leader_election.cc:302] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f24c1c43aa2456eb67b2cbb9ed1a948, dcc6733e9b3741578b9bc25604dd3bc6; no voters:
I20240501 14:11:27.473636 16134 raft_consensus.cc:2781] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:11:27.473698 16134 raft_consensus.cc:482] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:11:27.473748 16134 raft_consensus.cc:3037] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:27.474625 16134 raft_consensus.cc:504] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:27.474920 16134 leader_election.cc:288] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6 [CANDIDATE]: Term 1 election: Requested vote from peers 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411), 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132:40939)
I20240501 14:11:27.475216 15279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04aa763389aa4855babf3f9035177734" candidate_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948"
I20240501 14:11:27.475201 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04aa763389aa4855babf3f9035177734" candidate_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05"
I20240501 14:11:27.475306 15279 raft_consensus.cc:3037] T 04aa763389aa4855babf3f9035177734 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:27.475325 14886 raft_consensus.cc:3037] T 04aa763389aa4855babf3f9035177734 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:27.476027 15279 raft_consensus.cc:2454] T 04aa763389aa4855babf3f9035177734 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc6733e9b3741578b9bc25604dd3bc6 in term 1.
I20240501 14:11:27.476037 14886 raft_consensus.cc:2454] T 04aa763389aa4855babf3f9035177734 P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc6733e9b3741578b9bc25604dd3bc6 in term 1.
I20240501 14:11:27.476296 15083 leader_election.cc:302] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00035488ce0740898f6f92a508b01a05, dcc6733e9b3741578b9bc25604dd3bc6; no voters:
I20240501 14:11:27.476451 16134 raft_consensus.cc:2781] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:11:27.476567 16134 raft_consensus.cc:686] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 LEADER]: Becoming Leader. State: Replica: dcc6733e9b3741578b9bc25604dd3bc6, State: Running, Role: LEADER
I20240501 14:11:27.476706 16134 consensus_queue.cc:237] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:27.478257 14737 catalog_manager.cc:5484] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131). New cstate: current_term: 1 leader_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } }
I20240501 14:11:27.585026 16128 raft_consensus.cc:482] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:27.585148 16128 raft_consensus.cc:504] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:27.585515 16128 leader_election.cc:288] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411), dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131:39965)
I20240501 14:11:27.585923 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d37ef41bfb48cea9dcc05c45f782df" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05" is_pre_election: true
I20240501 14:11:27.585978 15148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d37ef41bfb48cea9dcc05c45f782df" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" is_pre_election: true
I20240501 14:11:27.586067 14886 raft_consensus.cc:2454] T 45d37ef41bfb48cea9dcc05c45f782df P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 0.
I20240501 14:11:27.586124 15148 raft_consensus.cc:2454] T 45d37ef41bfb48cea9dcc05c45f782df P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 0.
I20240501 14:11:27.586398 14952 leader_election.cc:302] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00035488ce0740898f6f92a508b01a05, 5f49c2edf1b54d83811340aaea48e682; no voters:
I20240501 14:11:27.586663 16128 raft_consensus.cc:2781] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:11:27.586712 16128 raft_consensus.cc:482] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:11:27.586753 16128 raft_consensus.cc:3037] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:27.587527 16128 raft_consensus.cc:504] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:27.587841 16128 leader_election.cc:288] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 election: Requested vote from peers 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411), dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131:39965)
I20240501 14:11:27.588116 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d37ef41bfb48cea9dcc05c45f782df" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05"
I20240501 14:11:27.588222 15148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d37ef41bfb48cea9dcc05c45f782df" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc6733e9b3741578b9bc25604dd3bc6"
I20240501 14:11:27.588274 14886 raft_consensus.cc:3037] T 45d37ef41bfb48cea9dcc05c45f782df P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:27.588344 15148 raft_consensus.cc:3037] T 45d37ef41bfb48cea9dcc05c45f782df P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:27.589095 14886 raft_consensus.cc:2454] T 45d37ef41bfb48cea9dcc05c45f782df P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 1.
I20240501 14:11:27.589346 15148 raft_consensus.cc:2454] T 45d37ef41bfb48cea9dcc05c45f782df P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 1.
I20240501 14:11:27.589481 14952 leader_election.cc:302] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00035488ce0740898f6f92a508b01a05, 5f49c2edf1b54d83811340aaea48e682; no voters:
I20240501 14:11:27.589699 16128 raft_consensus.cc:2781] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:11:27.589810 16128 raft_consensus.cc:686] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682 [term 1 LEADER]: Becoming Leader. State: Replica: 5f49c2edf1b54d83811340aaea48e682, State: Running, Role: LEADER
I20240501 14:11:27.589958 16128 consensus_queue.cc:237] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } }
I20240501 14:11:27.591826 14737 catalog_manager.cc:5484] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "5f49c2edf1b54d83811340aaea48e682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } }
W20240501 14:11:27.721289 16142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:11:27.721509 16142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:11:27.724407 16142 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 14:11:27.760931 16128 consensus_queue.cc:1035] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:27.775236 16088 consensus_queue.cc:1035] T d2405c8c944547f783912779f5c4d838 P 5f49c2edf1b54d83811340aaea48e682 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:27.809437 15148 raft_consensus.cc:1261] T 45d37ef41bfb48cea9dcc05c45f782df P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Refusing update from remote peer 5f49c2edf1b54d83811340aaea48e682: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:27.809787 14886 raft_consensus.cc:1261] T 45d37ef41bfb48cea9dcc05c45f782df P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Refusing update from remote peer 5f49c2edf1b54d83811340aaea48e682: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:27.809859 16088 consensus_queue.cc:1035] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:27.810194 16088 consensus_queue.cc:1035] T 45d37ef41bfb48cea9dcc05c45f782df P 5f49c2edf1b54d83811340aaea48e682 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:27.813239 15279 raft_consensus.cc:1261] T 04aa763389aa4855babf3f9035177734 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Refusing update from remote peer dcc6733e9b3741578b9bc25604dd3bc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:27.813550 14884 raft_consensus.cc:1261] T 04aa763389aa4855babf3f9035177734 P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Refusing update from remote peer dcc6733e9b3741578b9bc25604dd3bc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:11:27.813858 16134 consensus_queue.cc:1035] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:11:27.814148 16134 consensus_queue.cc:1035] T 04aa763389aa4855babf3f9035177734 P dcc6733e9b3741578b9bc25604dd3bc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 106.407 ms
time per row: 0.0532033 ms
I20240501 14:11:27.871551 14744 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41026:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:11:27.876380 15128 tablet_service.cc:1467] Processing CreateTablet for tablet 933a7c3f95784bf3b7cd1cadf5027087 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33749282f7ff4f859526454cabe1210b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:27.876480 15258 tablet_service.cc:1467] Processing CreateTablet for tablet 933a7c3f95784bf3b7cd1cadf5027087 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33749282f7ff4f859526454cabe1210b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:27.876590 14995 tablet_service.cc:1467] Processing CreateTablet for tablet 933a7c3f95784bf3b7cd1cadf5027087 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33749282f7ff4f859526454cabe1210b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:27.876688 15128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 933a7c3f95784bf3b7cd1cadf5027087. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:27.876763 15258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 933a7c3f95784bf3b7cd1cadf5027087. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:27.876844 14995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 933a7c3f95784bf3b7cd1cadf5027087. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:27.878423 15259 tablet_service.cc:1467] Processing CreateTablet for tablet dc7e40c90e03428eb9133668b7d7aed1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33749282f7ff4f859526454cabe1210b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:27.878646 15259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7e40c90e03428eb9133668b7d7aed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:27.878979 16200 tablet_bootstrap.cc:492] T 933a7c3f95784bf3b7cd1cadf5027087 P dcc6733e9b3741578b9bc25604dd3bc6: Bootstrap starting.
I20240501 14:11:27.879022 16199 tablet_bootstrap.cc:492] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682: Bootstrap starting.
I20240501 14:11:27.879220 16201 tablet_bootstrap.cc:492] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Bootstrap starting.
I20240501 14:11:27.879830 16200 tablet_bootstrap.cc:654] T 933a7c3f95784bf3b7cd1cadf5027087 P dcc6733e9b3741578b9bc25604dd3bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:27.880059 16201 tablet_bootstrap.cc:654] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:27.880152 16199 tablet_bootstrap.cc:654] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:27.880929 14866 tablet_service.cc:1467] Processing CreateTablet for tablet e428e074fa7647869820ff988a3d0779 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33749282f7ff4f859526454cabe1210b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:27.881107 15128 tablet_service.cc:1467] Processing CreateTablet for tablet dc7e40c90e03428eb9133668b7d7aed1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33749282f7ff4f859526454cabe1210b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:27.881181 14866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e428e074fa7647869820ff988a3d0779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:27.881162 15127 tablet_service.cc:1467] Processing CreateTablet for tablet e428e074fa7647869820ff988a3d0779 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33749282f7ff4f859526454cabe1210b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:27.881294 15128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7e40c90e03428eb9133668b7d7aed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:27.881393 15127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e428e074fa7647869820ff988a3d0779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:27.881783 16201 tablet_bootstrap.cc:492] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f24c1c43aa2456eb67b2cbb9ed1a948: No bootstrap required, opened a new log
I20240501 14:11:27.881858 16201 ts_tablet_manager.cc:1367] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:11:27.882277 16201 raft_consensus.cc:348] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:27.882375 16201 raft_consensus.cc:374] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:27.882407 16201 raft_consensus.cc:729] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f24c1c43aa2456eb67b2cbb9ed1a948, State: Initialized, Role: FOLLOWER
I20240501 14:11:27.882382 14997 tablet_service.cc:1467] Processing CreateTablet for tablet e428e074fa7647869820ff988a3d0779 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33749282f7ff4f859526454cabe1210b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:27.882530 16201 consensus_queue.cc:260] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:27.882578 14997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e428e074fa7647869820ff988a3d0779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:27.882795 16201 ts_tablet_manager.cc:1398] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:27.882963 16201 tablet_bootstrap.cc:492] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Bootstrap starting.
I20240501 14:11:27.883450 16205 tablet_bootstrap.cc:492] T e428e074fa7647869820ff988a3d0779 P 00035488ce0740898f6f92a508b01a05: Bootstrap starting.
I20240501 14:11:27.883800 16201 tablet_bootstrap.cc:654] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:27.884322 16205 tablet_bootstrap.cc:654] T e428e074fa7647869820ff988a3d0779 P 00035488ce0740898f6f92a508b01a05: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:27.884801 16201 tablet_bootstrap.cc:492] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948: No bootstrap required, opened a new log
I20240501 14:11:27.884883 16201 ts_tablet_manager.cc:1367] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:11:27.885280 16201 raft_consensus.cc:348] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:27.885327 16205 tablet_bootstrap.cc:492] T e428e074fa7647869820ff988a3d0779 P 00035488ce0740898f6f92a508b01a05: No bootstrap required, opened a new log
I20240501 14:11:27.885372 16201 raft_consensus.cc:374] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:27.885408 16205 ts_tablet_manager.cc:1367] T e428e074fa7647869820ff988a3d0779 P 00035488ce0740898f6f92a508b01a05: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:11:27.885411 16201 raft_consensus.cc:729] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f24c1c43aa2456eb67b2cbb9ed1a948, State: Initialized, Role: FOLLOWER
I20240501 14:11:27.885557 16201 consensus_queue.cc:260] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:27.885813 16205 raft_consensus.cc:348] T e428e074fa7647869820ff988a3d0779 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:27.885932 16205 raft_consensus.cc:374] T e428e074fa7647869820ff988a3d0779 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:27.885967 16205 raft_consensus.cc:729] T e428e074fa7647869820ff988a3d0779 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00035488ce0740898f6f92a508b01a05, State: Initialized, Role: FOLLOWER
I20240501 14:11:27.886090 16205 consensus_queue.cc:260] T e428e074fa7647869820ff988a3d0779 P 00035488ce0740898f6f92a508b01a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:27.886219 14995 tablet_service.cc:1467] Processing CreateTablet for tablet dc7e40c90e03428eb9133668b7d7aed1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33749282f7ff4f859526454cabe1210b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:11:27.886346 16205 ts_tablet_manager.cc:1398] T e428e074fa7647869820ff988a3d0779 P 00035488ce0740898f6f92a508b01a05: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:11:27.886420 14995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7e40c90e03428eb9133668b7d7aed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:11:27.887274 16201 ts_tablet_manager.cc:1398] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:11:27.888522 16199 tablet_bootstrap.cc:492] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682: No bootstrap required, opened a new log
I20240501 14:11:27.888604 16199 ts_tablet_manager.cc:1367] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:11:27.889025 16200 tablet_bootstrap.cc:492] T 933a7c3f95784bf3b7cd1cadf5027087 P dcc6733e9b3741578b9bc25604dd3bc6: No bootstrap required, opened a new log
I20240501 14:11:27.889104 16200 ts_tablet_manager.cc:1367] T 933a7c3f95784bf3b7cd1cadf5027087 P dcc6733e9b3741578b9bc25604dd3bc6: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 14:11:27.889096 16199 raft_consensus.cc:348] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:27.889182 16199 raft_consensus.cc:374] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:27.889218 16199 raft_consensus.cc:729] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f49c2edf1b54d83811340aaea48e682, State: Initialized, Role: FOLLOWER
I20240501 14:11:27.889333 16199 consensus_queue.cc:260] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:27.889513 16200 raft_consensus.cc:348] T 933a7c3f95784bf3b7cd1cadf5027087 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:27.889597 16200 raft_consensus.cc:374] T 933a7c3f95784bf3b7cd1cadf5027087 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:27.889596 16199 ts_tablet_manager.cc:1398] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:11:27.889628 16200 raft_consensus.cc:729] T 933a7c3f95784bf3b7cd1cadf5027087 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc6733e9b3741578b9bc25604dd3bc6, State: Initialized, Role: FOLLOWER
I20240501 14:11:27.889758 16200 consensus_queue.cc:260] T 933a7c3f95784bf3b7cd1cadf5027087 P dcc6733e9b3741578b9bc25604dd3bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:27.889806 16199 tablet_bootstrap.cc:492] T e428e074fa7647869820ff988a3d0779 P 5f49c2edf1b54d83811340aaea48e682: Bootstrap starting.
I20240501 14:11:27.890030 16200 ts_tablet_manager.cc:1398] T 933a7c3f95784bf3b7cd1cadf5027087 P dcc6733e9b3741578b9bc25604dd3bc6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:27.890192 16200 tablet_bootstrap.cc:492] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6: Bootstrap starting.
I20240501 14:11:27.890640 16199 tablet_bootstrap.cc:654] T e428e074fa7647869820ff988a3d0779 P 5f49c2edf1b54d83811340aaea48e682: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:27.891044 16200 tablet_bootstrap.cc:654] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:27.891757 16199 tablet_bootstrap.cc:492] T e428e074fa7647869820ff988a3d0779 P 5f49c2edf1b54d83811340aaea48e682: No bootstrap required, opened a new log
I20240501 14:11:27.891875 16199 ts_tablet_manager.cc:1367] T e428e074fa7647869820ff988a3d0779 P 5f49c2edf1b54d83811340aaea48e682: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:11:27.892275 16199 raft_consensus.cc:348] T e428e074fa7647869820ff988a3d0779 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:27.892369 16199 raft_consensus.cc:374] T e428e074fa7647869820ff988a3d0779 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:27.892398 16199 raft_consensus.cc:729] T e428e074fa7647869820ff988a3d0779 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f49c2edf1b54d83811340aaea48e682, State: Initialized, Role: FOLLOWER
I20240501 14:11:27.892522 16199 consensus_queue.cc:260] T e428e074fa7647869820ff988a3d0779 P 5f49c2edf1b54d83811340aaea48e682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:27.892809 16199 ts_tablet_manager.cc:1398] T e428e074fa7647869820ff988a3d0779 P 5f49c2edf1b54d83811340aaea48e682: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:11:27.893024 16199 tablet_bootstrap.cc:492] T dc7e40c90e03428eb9133668b7d7aed1 P 5f49c2edf1b54d83811340aaea48e682: Bootstrap starting.
I20240501 14:11:27.893733 16199 tablet_bootstrap.cc:654] T dc7e40c90e03428eb9133668b7d7aed1 P 5f49c2edf1b54d83811340aaea48e682: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:27.894265 16200 tablet_bootstrap.cc:492] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6: No bootstrap required, opened a new log
I20240501 14:11:27.894330 16200 ts_tablet_manager.cc:1367] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:11:27.894672 16200 raft_consensus.cc:348] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:27.894747 16200 raft_consensus.cc:374] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:27.894765 16200 raft_consensus.cc:729] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc6733e9b3741578b9bc25604dd3bc6, State: Initialized, Role: FOLLOWER
I20240501 14:11:27.894757 16199 tablet_bootstrap.cc:492] T dc7e40c90e03428eb9133668b7d7aed1 P 5f49c2edf1b54d83811340aaea48e682: No bootstrap required, opened a new log
I20240501 14:11:27.894832 16199 ts_tablet_manager.cc:1367] T dc7e40c90e03428eb9133668b7d7aed1 P 5f49c2edf1b54d83811340aaea48e682: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:11:27.894852 16200 consensus_queue.cc:260] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:27.895076 16200 ts_tablet_manager.cc:1398] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:11:27.895197 16200 tablet_bootstrap.cc:492] T dc7e40c90e03428eb9133668b7d7aed1 P dcc6733e9b3741578b9bc25604dd3bc6: Bootstrap starting.
I20240501 14:11:27.895231 16199 raft_consensus.cc:348] T dc7e40c90e03428eb9133668b7d7aed1 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:27.895314 16199 raft_consensus.cc:374] T dc7e40c90e03428eb9133668b7d7aed1 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:27.895344 16199 raft_consensus.cc:729] T dc7e40c90e03428eb9133668b7d7aed1 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f49c2edf1b54d83811340aaea48e682, State: Initialized, Role: FOLLOWER
I20240501 14:11:27.895442 16199 consensus_queue.cc:260] T dc7e40c90e03428eb9133668b7d7aed1 P 5f49c2edf1b54d83811340aaea48e682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:27.895748 16199 ts_tablet_manager.cc:1398] T dc7e40c90e03428eb9133668b7d7aed1 P 5f49c2edf1b54d83811340aaea48e682: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:11:27.896207 16200 tablet_bootstrap.cc:654] T dc7e40c90e03428eb9133668b7d7aed1 P dcc6733e9b3741578b9bc25604dd3bc6: Neither blocks nor log segments found. Creating new log.
I20240501 14:11:27.897255 16200 tablet_bootstrap.cc:492] T dc7e40c90e03428eb9133668b7d7aed1 P dcc6733e9b3741578b9bc25604dd3bc6: No bootstrap required, opened a new log
I20240501 14:11:27.897331 16200 ts_tablet_manager.cc:1367] T dc7e40c90e03428eb9133668b7d7aed1 P dcc6733e9b3741578b9bc25604dd3bc6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:11:27.897702 16200 raft_consensus.cc:348] T dc7e40c90e03428eb9133668b7d7aed1 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:27.897786 16200 raft_consensus.cc:374] T dc7e40c90e03428eb9133668b7d7aed1 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:11:27.897814 16200 raft_consensus.cc:729] T dc7e40c90e03428eb9133668b7d7aed1 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc6733e9b3741578b9bc25604dd3bc6, State: Initialized, Role: FOLLOWER
I20240501 14:11:27.897928 16200 consensus_queue.cc:260] T dc7e40c90e03428eb9133668b7d7aed1 P dcc6733e9b3741578b9bc25604dd3bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:27.898234 16200 ts_tablet_manager.cc:1398] T dc7e40c90e03428eb9133668b7d7aed1 P dcc6733e9b3741578b9bc25604dd3bc6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:11:27.916497 16054 raft_consensus.cc:482] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:27.916592 16054 raft_consensus.cc:504] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:27.916930 16054 leader_election.cc:288] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131:39965), 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130:40469)
I20240501 14:11:27.917208 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7e40c90e03428eb9133668b7d7aed1" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f49c2edf1b54d83811340aaea48e682" is_pre_election: true
I20240501 14:11:27.917208 15148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7e40c90e03428eb9133668b7d7aed1" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" is_pre_election: true
I20240501 14:11:27.917317 15148 raft_consensus.cc:2454] T dc7e40c90e03428eb9133668b7d7aed1 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in term 0.
I20240501 14:11:27.917335 15016 raft_consensus.cc:2454] T dc7e40c90e03428eb9133668b7d7aed1 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in term 0.
I20240501 14:11:27.917579 15214 leader_election.cc:302] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f24c1c43aa2456eb67b2cbb9ed1a948, dcc6733e9b3741578b9bc25604dd3bc6; no voters:
I20240501 14:11:27.917768 16054 raft_consensus.cc:2781] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:11:27.917832 16054 raft_consensus.cc:482] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:11:27.917874 16054 raft_consensus.cc:3037] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:27.918709 16054 raft_consensus.cc:504] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:27.919054 16054 leader_election.cc:288] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 election: Requested vote from peers dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131:39965), 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130:40469)
I20240501 14:11:27.919360 15148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7e40c90e03428eb9133668b7d7aed1" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc6733e9b3741578b9bc25604dd3bc6"
I20240501 14:11:27.919482 15148 raft_consensus.cc:3037] T dc7e40c90e03428eb9133668b7d7aed1 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:27.919456 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7e40c90e03428eb9133668b7d7aed1" candidate_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f49c2edf1b54d83811340aaea48e682"
I20240501 14:11:27.919553 15016 raft_consensus.cc:3037] T dc7e40c90e03428eb9133668b7d7aed1 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:27.920428 15148 raft_consensus.cc:2454] T dc7e40c90e03428eb9133668b7d7aed1 P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in term 1.
I20240501 14:11:27.920428 15016 raft_consensus.cc:2454] T dc7e40c90e03428eb9133668b7d7aed1 P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f24c1c43aa2456eb67b2cbb9ed1a948 in term 1.
I20240501 14:11:27.920678 15214 leader_election.cc:302] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f24c1c43aa2456eb67b2cbb9ed1a948, dcc6733e9b3741578b9bc25604dd3bc6; no voters:
I20240501 14:11:27.920938 16054 raft_consensus.cc:2781] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:11:27.921103 16054 raft_consensus.cc:686] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 LEADER]: Becoming Leader. State: Replica: 5f24c1c43aa2456eb67b2cbb9ed1a948, State: Running, Role: LEADER
I20240501 14:11:27.921238 16054 consensus_queue.cc:237] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } }
I20240501 14:11:27.922641 14744 catalog_manager.cc:5484] T dc7e40c90e03428eb9133668b7d7aed1 P 5f24c1c43aa2456eb67b2cbb9ed1a948 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132). New cstate: current_term: 1 leader_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } }
I20240501 14:11:27.993350 16088 raft_consensus.cc:482] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:27.993463 16088 raft_consensus.cc:504] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:27.993814 16088 leader_election.cc:288] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131:39965), 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132:40939)
I20240501 14:11:27.994171 15148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933a7c3f95784bf3b7cd1cadf5027087" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" is_pre_election: true
I20240501 14:11:27.994196 15279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933a7c3f95784bf3b7cd1cadf5027087" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" is_pre_election: true
I20240501 14:11:27.994292 15148 raft_consensus.cc:2454] T 933a7c3f95784bf3b7cd1cadf5027087 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 0.
I20240501 14:11:27.994323 15279 raft_consensus.cc:2454] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 0.
I20240501 14:11:27.994575 14952 leader_election.cc:302] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f49c2edf1b54d83811340aaea48e682, dcc6733e9b3741578b9bc25604dd3bc6; no voters:
I20240501 14:11:27.994758 16088 raft_consensus.cc:2781] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:11:27.994812 16088 raft_consensus.cc:482] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:11:27.994863 16088 raft_consensus.cc:3037] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:27.995687 16088 raft_consensus.cc:504] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:27.996042 16088 leader_election.cc:288] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 election: Requested vote from peers dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131:39965), 5f24c1c43aa2456eb67b2cbb9ed1a948 (127.22.252.132:40939)
I20240501 14:11:27.996353 15148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933a7c3f95784bf3b7cd1cadf5027087" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc6733e9b3741578b9bc25604dd3bc6"
I20240501 14:11:27.996383 15279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933a7c3f95784bf3b7cd1cadf5027087" candidate_uuid: "5f49c2edf1b54d83811340aaea48e682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948"
I20240501 14:11:27.996469 15148 raft_consensus.cc:3037] T 933a7c3f95784bf3b7cd1cadf5027087 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:27.996474 15279 raft_consensus.cc:3037] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:27.997347 15279 raft_consensus.cc:2454] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f24c1c43aa2456eb67b2cbb9ed1a948 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 1.
I20240501 14:11:27.997347 15148 raft_consensus.cc:2454] T 933a7c3f95784bf3b7cd1cadf5027087 P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f49c2edf1b54d83811340aaea48e682 in term 1.
I20240501 14:11:27.997611 14953 leader_election.cc:302] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f24c1c43aa2456eb67b2cbb9ed1a948, 5f49c2edf1b54d83811340aaea48e682; no voters:
I20240501 14:11:27.997805 16088 raft_consensus.cc:2781] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:11:27.997890 16088 raft_consensus.cc:686] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682 [term 1 LEADER]: Becoming Leader. State: Replica: 5f49c2edf1b54d83811340aaea48e682, State: Running, Role: LEADER
I20240501 14:11:27.998067 16088 consensus_queue.cc:237] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:27.999651 14744 catalog_manager.cc:5484] T 933a7c3f95784bf3b7cd1cadf5027087 P 5f49c2edf1b54d83811340aaea48e682 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130). New cstate: current_term: 1 leader_uuid: "5f49c2edf1b54d83811340aaea48e682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f24c1c43aa2456eb67b2cbb9ed1a948" member_type: VOTER last_known_addr { host: "127.22.252.132" port: 40939 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } }
I20240501 14:11:28.102169 15882 raft_consensus.cc:482] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:28.102288 15882 raft_consensus.cc:504] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:28.102717 15882 leader_election.cc:288] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411), 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130:40469)
I20240501 14:11:28.103123 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e428e074fa7647869820ff988a3d0779" candidate_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f49c2edf1b54d83811340aaea48e682" is_pre_election: true
I20240501 14:11:28.103123 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e428e074fa7647869820ff988a3d0779" candidate_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05" is_pre_election: true
I20240501 14:11:28.103283 15016 raft_consensus.cc:2454] T e428e074fa7647869820ff988a3d0779 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc6733e9b3741578b9bc25604dd3bc6 in term 0.
I20240501 14:11:28.103283 14884 raft_consensus.cc:2454] T e428e074fa7647869820ff988a3d0779 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc6733e9b3741578b9bc25604dd3bc6 in term 0.
I20240501 14:11:28.103560 15081 leader_election.cc:302] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f49c2edf1b54d83811340aaea48e682, dcc6733e9b3741578b9bc25604dd3bc6; no voters:
I20240501 14:11:28.103760 15882 raft_consensus.cc:2781] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:11:28.103814 15882 raft_consensus.cc:482] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:11:28.103848 15882 raft_consensus.cc:3037] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:28.104780 15882 raft_consensus.cc:504] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:28.105103 15882 leader_election.cc:288] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6 [CANDIDATE]: Term 1 election: Requested vote from peers 00035488ce0740898f6f92a508b01a05 (127.22.252.129:33411), 5f49c2edf1b54d83811340aaea48e682 (127.22.252.130:40469)
I20240501 14:11:28.105484 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e428e074fa7647869820ff988a3d0779" candidate_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f49c2edf1b54d83811340aaea48e682"
I20240501 14:11:28.105587 15016 raft_consensus.cc:3037] T e428e074fa7647869820ff988a3d0779 P 5f49c2edf1b54d83811340aaea48e682 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:28.105724 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e428e074fa7647869820ff988a3d0779" candidate_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00035488ce0740898f6f92a508b01a05"
I20240501 14:11:28.105849 14884 raft_consensus.cc:3037] T e428e074fa7647869820ff988a3d0779 P 00035488ce0740898f6f92a508b01a05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:11:28.106388 15016 raft_consensus.cc:2454] T e428e074fa7647869820ff988a3d0779 P 5f49c2edf1b54d83811340aaea48e682 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc6733e9b3741578b9bc25604dd3bc6 in term 1.
I20240501 14:11:28.106621 14884 raft_consensus.cc:2454] T e428e074fa7647869820ff988a3d0779 P 00035488ce0740898f6f92a508b01a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc6733e9b3741578b9bc25604dd3bc6 in term 1.
I20240501 14:11:28.106662 15081 leader_election.cc:302] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f49c2edf1b54d83811340aaea48e682, dcc6733e9b3741578b9bc25604dd3bc6; no voters:
I20240501 14:11:28.106832 15882 raft_consensus.cc:2781] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:11:28.106931 15882 raft_consensus.cc:686] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6 [term 1 LEADER]: Becoming Leader. State: Replica: dcc6733e9b3741578b9bc25604dd3bc6, State: Running, Role: LEADER
I20240501 14:11:28.107088 15882 consensus_queue.cc:237] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } }
I20240501 14:11:28.108867 14744 catalog_manager.cc:5484] T e428e074fa7647869820ff988a3d0779 P dcc6733e9b3741578b9bc25604dd3bc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcc6733e9b3741578b9bc25604dd3bc6 (127.22.252.131). New cstate: current_term: 1 leader_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00035488ce0740898f6f92a508b01a05" member_type: VOTER last_known_addr { host: "127.22.252.129" port: 33411 } } peers { permanent_uuid: "dcc6733e9b3741578b9bc25604dd3bc6" member_type: VOTER last_known_addr { host: "127.22.252.131" port: 39965 } } peers { permanent_uuid: "5f49c2edf1b54d83811340aaea48e682" member_type: VOTER last_known_addr { host: "127.22.252.130" port: 40469 } } }
W20240501 14:11:28.272922 16216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:11:28.273130 16216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:11:28.276072 16216 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'